DISCO: Describing Images Using Scene Contexts and Objects

Ifeoma NwoguUniversity of Rochester,

Rochester, NY 14627

Yingbo ZhouUniversity at Buffalo, SUNY

Buffalo, NY 14260

Christopher BrownUniversity of Rochester,

Rochester, NY 14627

Abstract

In this paper, we propose a bottom-up approach to gen-erating short descriptive sentences from images, to en-hance scene understanding. We demonstrate automaticmethods for mapping the visual content in an image tonatural spoken or written language. We also introducea human-in-the-loop evaluation strategy that quantita-tively captures the meaningfulness of the generated sen-tences. We recorded a correctness rate of 60.34% whenhuman users were asked to judge the meaningfulness ofthe sentences generated from relatively challenging im-ages. Also, our automatic methods compared well withthe state-of-the-art techniques for the related computervision tasks.

1 Introduction

The ability to have a machine correctly parse, recognize andcommunicate naturally about the visual world around is asignificant advancement in machine intelligence, which tac-itly is an augmentation of the human intellect. For real-lifescenes, humans can communicate a concise description inthe form of a sentence relatively easily. Such descriptionsmight identify the most interesting objects, what they aredoing, and/or where this is happening. These descriptionsare rich, accurate, and in good agreement between other hu-mans. They are concise: much is omitted, because humanstend not to mention objects or events that they judge to beless significant.

Similar to how humans use sentences to communicate,we present naturalistic machine generated sentences, whichwhen presented to humans can be used to identify specificscenes being described. As an example, the text below isa sentence generated from our system, DISCO (DescribingImages using Scene Contexts and Objects). The reader is en-couraged to select one of the four images presented in Figure1 that is best described by the sentences below.

The image scenery is set with a view of tall buildings;

There are at least 5 people in the picture. Someone in

the picture has on something bluish-purple. The people

are standing on the ground. The light gray buildings are

quite prominent in the scenery.

Copyright c© 2011, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

Our descriptions apply to general, all-purpose outdoor im-ages. These are significantly more challenging to describethan images in a specific domain such as street scenes, mar-itime scenes etc, since there are very few domain-specificconstraints that can be applied to aid scene understand-ing. Included in our data set are images taken in natu-ral light, at dusk and at dawn, in foggy weather, in snowstorms and at night. Also included in the data set are im-ages taken at different viewpoints. Unlike the images inmany of the scene categorization benchmark datasets, whichtypically only contain “pure scenes”, our dataset containssingle images that simultaneously belong to multiple scenecategories, and taken at multiple viewpoints. In many ofthese images, everything potentially co-occurs with every-thing else. Some examples from our test dataset are shownin Figure 1. The complete results of vision-to-language de-scriptor can be accessed athttp://www.zmldesign.com/SUnProject 1.

Figure 1: Examples of images from our sentence generationdataset.

1.1 Physically grounded application scenarios

Scenario 1: Travel Companion Bot. An example ofan application of the technique we propose is a scene-conversation-system deployed in a car, having the ability

1Web address is case-sensitive. The reader is encouraged to testand fill in richer descriptions to help advance this research

Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence

1487

to converse with its driver concerning its surrounding out-door environment. Knight Rider was an American televi-sion series that originally ran from 1982 to 1986, involv-ing a sapient, artificial intelligence-based, talking car, KITT.Implementing the technology to realize some of the mecha-nisms manifested in the fictitious robot, KITT, such as beingable to “understand and communicate about physical visualsurrounding” would be an advancement in artificial intelli-gence. Since cars do not remain only within a specific scenecategory, but can be driven within any outdoor scene such asfor a picturesque family vacation trip, to the beach, withinthe city, etc., having an all-purpose scene-independent im-age description generator is not far-fetched in such a con-text. Such a car/robot would need to understood its surround-ings and be able to communicate in a natural language withits driver and passengers. The conversation could be high-level general scene descriptions, or could be semantically“zoomed-in”, to be specific about an object or an event inthe scene.

Scenario 2 - Safety patrol Bot. This scenario involvesautonomous robots patrolling highways, beaches, countryroads, oil rigs, etc., with a variety of sensors to capturevideos and sound. By providing such robots with the abil-ity to speak in a natural language, a large amount of visualinformation can be compressed via language and transferredat significantly lower bandwidth, to a human supervisor, andpotentially to other patrol bots in the area.

DISCO is currently in the early stages of realizing suchphysically grounded robots, by providing rich, descriptivesentences about physical environments.

�������������� ���

����� ����� ���

�� �����������

��� ��������� ���

����� ����� ��� ���

������� ������� ���

��� ���� ��� ���

�������������� �� �� ���

����������

Figure 2: End-to-end process flow of our visual content ex-traction and sentence generation approach

1.2 Overview of our approach

Figure 2 presents an overview of our visual content extrac-tion and sentence generation process which is initiated withan original input image presented to our system DISCO. Theglobal image properties - scene categorization and lightingestimation are first performed on the entire image. Imagesegments are then computed and presented to the multi-classlabeling and geometric layout estimation modules in orderto generate the specific details of the image. We pre-definedan ontology of outdoor images consisting of placeholdersfor scene categories, foreground objects, lighting conditions,background regions and surface layouts. Sentences about theimage are constructed from the specific instances of the data

types in the ontology, using a forward chaining inference en-gine.

1.3 Contributions

We summarize our primary contributions as (i) an automatedend-to-end technique for generating natural language de-scriptions from the visual content in images, that can bepotentially useful in different real-world applications; (ii)defining a novel computer vision problem of classifying im-age illumination; (iii) a novel real-time technique for select-ing optimal regions in, and parsing ad-hoc outdoor images.We show that our technique compares well with the state-of-the-art methods on public datasets; and (iv) a quantitativeevaluation technique with humans-in-the-loop, for measur-ing the accuracy and meaningfulness of generated image de-scriptions. Unlike typical tests in computer vision which areinitiated by humans (e.g. a person presents an input imageand the system returns the image segments or edges), thistest is initiated by the computer and a human judge is ex-pected to respond to measure of the meaningfulness of theproposed image description; The advantage of this techniqueis its quantitative measure of sentence meaningfulness.

2 Related Work

Sentence generation from images touches on many facetsof computer vision that have previously been treated asseparate problems. The first of these, image segmentation,is primarily concerned with perceptually grouping imageparts from lower-level primitive image features, withoutany knowledge of the image content, in order to obtainmeaningful higher-level structures. Image segmentation hasbeen addressed extensively in the computer vision litera-ture, including thresholding techniques, edge-based meth-ods, connectivity-preserving relaxation methods (such asactive contours and level set techniques) and region-basedtechniques. In this study we are only concerned with region-based segmentation which involves partitioning an imageinto connected regions by grouping neighboring pixels ofsimilar feature values (or cues). These include intensity,color, texture, motion, etc. Many of the advancements inregion-based image segmentation are therefore either im-provements in cue selection (also referred to as model orderselection) such as (Ren, Fowlkes, and Malik 2005), and im-provements in optimal cue combination (Alpert et al. 2007),(Shi and Malik 2000), (Comaniciu and Meer 2002). A studyby (Rabinovich et al. 2006) proposed a framework that com-bined both techniques and as an output presented the userwith a continuum of stable segmentations, rather than justone solution. A stable clustering is one that is repeatableover several perturbations.

Multi-class image labeling, the next facet we review,has been successfully addressed by a number of researchworks; (Gould, Fulton, and Koller 2009) present an exten-sive overview of region-based segmentation and labelingtechniques in the “background and related work” section oftheir paper. The goal here in image labeling is to assign ev-ery image pixel a single class label. Usually the techniqueinvolves constructing a Conditional Random Field (CRF)

1488

over either pixels or small coherent regions, superpixels. Anappearance based unary potential and a pairwise smoothingpotential is defined to encourage neighboring pixels to takethe same class label. (Gould, Fulton, and Koller 2009) pro-pose a labeling technique using multiple stable segmenta-tions as input and define an energy minimization techniqueover several aspects of the scene, including geometric anddepth estimations. Other works such as by (Li, Socher, andLi 2009), (Tu et al. 2005) have developed unified methodsto simultaneously segment, label and/or categorize images.

Lastly, in the area of sentence generation from general im-age data (not domain specific images such as sports) there islittle reported research. (Farhadi et al. 2010) generated sen-tences from images, using a model that assumed a space ofMeanings. A score is obtained by comparing an estimate ofmeaning obtained from an image to one obtained from a sen-tence. Each estimate of meaning comes from a discrimina-tive procedure that is learned from training data. Althoughthis model is very flexible in its design and yielded visu-ally pleasing results, it is not obvious how well their de-fined meanings space corresponds to human understanding,the ultimate goal in our proposed technique. As the authorsstated in the paper, their “underlying estimate of meaningis impoverished”. The technique though, shows tremendouspromise for text-based, query-driven image retrieval andhigh-level object detection from images, the other side ofour human-to-computer communication goal, unaddressedin this paper.

Another example of research in sentence generation fromimages was reported by (Gupta et al. 2009) who gener-ated narrations relating to a sports event in videos, usinga compositional model of AND-OR graphs. Their researchshowed very encouraging results in the specific domain andthe structures of expected events were known beforehand,which greatly enhanced their sentence generation as wellas the evaluation of results. (Yao et al. 2010) also gener-ated sentences from temporal sequences in pre-specified do-mains (surveillance in an urban traffic scene and in a mar-itime scene), also using an AND-OR graph over a visualknowledge representation. Although they demonstrated theiralgorithms within these specific domains in their paper, noevaluation for sentence correctness was provided.

3 Visual Content Extraction

Image segmentation Our image segmentation paradigm isinitiated with the generation of superpixels j = 1, . . . N . Im-age gradient magnitudes were used as inputs to the water-shed algorithm (as implemented in Matlab image process-ing toolkit) and the resulting superpixels are highly regularin shape and size. Our segmentation algorithm first sorts theresulting superpixels in increasing order of f(·), where fin our case is a single value computed by bit-shifting theaverage color in a superpixel. The order is traversed onceand sorting, the most expensive computation runs in orderof O(n log n) Each region in the sorting order will be com-bined with its neighbors based on a merging predicate given

by:

P (Ri, RN (i)) =

{true iff |Ri − RN(i)| ≥ T ,false otherwise

where Ri is a region (containing one or more alreadymerged superpixels) and RN (i) is an unmerged neighbor ofRi; Ri is a vector representing the aggregated cues in regionRi. The aggregation in each cue is described in the list be-low. T is a vector of threshold values, where each dimensionof T is a cue threshold value, learned from training data. Theoperator | · | represents the χ2 distance.

The feature list used in our merging predicate are:

• HSV colors: represented by histogram distributions. Theaggregate value is the histogram re-distribution of themerged region.

• Texture: represented by the mean absolute responses andhistogram responses of 4-scale, 6-orientation LM fil-ters. To compute aggregate values, the unnormalized his-tograms are summed. After the merge is completed, thetexture histograms are normalized.

Multiple segmentations can be encouraged by varying thethreshold values so that stable segmentations can be ex-tracted. But in our experiments, by using multiple segmen-tations, our pixelwise accuracy dropped from about 79% to70%, hence we opted to use only a single segmentation.

Scene categorization For scene categorization, we clas-sify the GIST description (Oliva and Torralba 2001) of theimage, performing a one-versus-all test with the rbf kernelon a support vector machine. A 64-dimensional feature vec-tor is computed over the image is used for classification, re-sulting in a class C, where C ∈ {Mountain, Woods, Beach,Street, Highway, Neighborhood-suburb, Neighborhood-city,Garden, Tall buildings, Plains}.

These categories match well to those defined in (Oliva andTorralba 2001), with the exception of the additional classes.Also, we greatly modified their scene categorization datasetby removing much of the “pure scene” images and includedimages with mixed scenes (such as a highway by the ocean,or tall buildings surrounded by open plains), for training andtesting. In updating the scene training data to more realisticimages, the accuracy rate dropped from 82% as reported bythe authors to 68%.

Scene illumination using location probability mapsThe scene illumination is computed by training locationprobability maps over images belonging to the different illu-mination classes. A location probability map divides everyimage in our training set into a fixed number of horizontalslices to coarsely capture its location properties. Each sliceis converted to the Red-Green, Blue-Yellow color space and16-bin color histograms are computed for each channel ineach slice. The total image feature is therefore a concate-nation of the histogram data in a fixed order. Auto-logisticregression is used for classification, resulting in a class L,where L ∈ natural light, Sunset, Nighttime, Foggy. The R-Gand B-Y conversion is given by:

1489

(O1

O2

O3

)=

(5.2 −4.9 1.7

−1.3 −1.9 3.01.0 0.0 1.0

)(0.7R1.1G1.0B

)

O1 is the R-G channel while O2 is the B-Y channel. Aftertesting several other color spaces, we found this to be themost discriminant of scene illumination, with its sensitivityto red-green and blue-yellow separately.

Spatial geometry recovery and object detections Weuse the model by (Hoiem, Efros, and Hebert 2007) for es-timating surface layouts and the object detectors by (Felzen-szwalb et al. 2010) to detect foreground objects. The objects-of-interest in DISCOare people, cars, trains, motorbikes,buses, cows and boats

3.1 Fusing scene categories, multi-class regionlabels and geometric locations

Our region labeling aims to assign a label to each segmentsin the image. We assume that the segment resolution is suf-ficient to capture all regions of interest in the image. Eachsegment is labeled by initially assigning a class to it, basedsolely on its appearance. Node potentials are based on thepre-computed appearance features, and they include the ab-solute differences of the mean RGB, L*a*b*, HSV, texturefilter responses and x-y location values of the segment cen-troid, the symmetrized Kullback-Leibler divergence of thecolor and texture histograms and perspective differences.The ensuing region labels are assigned at the segment levelas Rj .

Our fusion model seeks to improve the overall estimatedscene class, region and geometry labels and object detection.We accomplish this by incorporating both scene-based andobject based context into our final inference.

Figure 3 shows the simplified graphical model of our fu-sion technique, where the graph consists of two segmentsSm and Sn with feature vectors with features Xm and Xn

respectively. The joint distribution of the model can be writ-

�

��

��

��

��

Figure 3: The simplified fusion graphical model showing thefactorization for an image with only two segments

ten as:

P (C,R1...N , G1...N |X1...N ) =N∏i=1

ψi

N∏j=1

φj (1)

where ψi = ψ(Ri, Gi, Xi) is a clique potential, andφj = φ(Rj , C).

The 3-clique is factorized into pairwise potentials for easeof computations.

ψ(Ri, Gi, Xi) ∝ p(Ri|Xi)p(Gi|Xi)p(Ri, Gi) (2)

The first two elements on the RHS of Equation 2 were pre-computed during the independent label classification.p(Ri, Gi) is represented by a Region-Location matrix com-puted during training where the number of co-occurrences ofregions and geometrical labels were counted, recorded andnormalized. Similarly, φ(Rj , C) is derived by computing theco-occurrences of regions and scene categories.

4 Sentence Generation

Given the visual content extracted from an image, it is pos-sible to generate sentences about the image using forwardchaining production rules. The system consists of the pre-defined set of rules, a working memory to store the tempo-rary data and the forward chaining inference.

Input: original image, empty string-finalstr, C, L, {R}, {G},{bboxes}

Output: Populated string-finalstrGenerate a static color palette of 26 colors;Randomly select from a list of scene lighting sentencesPopulate the scene lighting sentence with token L;Append new sentence to finalstrRandomly select from a list of scene category sentencesPopulate the scene category sentence with token C;Append new sentence to finalstrforeach object category j do

foreach bboxes i doif Score(i,j) ≥ threshold then

if L = ’Natural lighting’ or ’Foggy’ thenCompute foreground object color;Populate object sentence with number andcolors of objects j;

endelse

Populate object sentence with number ofobjects j;

endAppend new sentence to finalstr;

end

end

endif L = ’Natural lighting’ or ’Foggy’ then

Populate templates with region information in the ground,vertical and sky planes from {R} and {G};Compute background region colors;Append new sentences to finalstr;

end

Algorithm 1: Sentence generation from image content

The output of the visual content extraction process foran image include: (i) the most likely scene category C se-lected from the set of nine category labels; (ii) the scenelighting category L selected from the set of four labels; (iii)the set of background regions, {R}; (iv) the set of geometricregions{G}; and (v) the set of foreground object boundingboxes bboxes, for seven object categories. Color estimations

1490

for objects or regions are performed in the L*a*b* space us-ing Euclidean distances between the object/region medianvalue and the pre-defined 26-color palette.

The sentence generation is summarized in Algorithm 1

5 Experiments and Results

Due to the multi-faceted nature of our work, we conductedmultiple experiments to evaluate the performance of theintermediary computer vision sub-processes, and one finalhuman-in-the-loop experiment to evaluate the meaningful-ness of the generated sentences.

Scene Categorization The scene categorization algo-rithm was initially presented by Torralba et. al. (Oliva andTorralba 2001), and the “pure scene” dataset presented hassince been extended by several other researchers including(Li, Socher, and Li 2009). We discarded the indoor labelsand used only the outdoor ones. We extended the train-ing/validation dataset by introducing several real-life imagescontaining multiple scenes and objects. Figure 4 shows theresults of running the algorithm both on the “pure scene”dataset, and on the more realistic diverse dataset that weextended. Although the cross-validation accuracy droppedfrom about 80% to 68% after the dataset was extended, itsperformance on our final more realistic testing dataset im-proved.

Figure 4: The confusion matrix on the left shows our algo-rithm using the pure scene data, while the right is the samealgorithm on the more realistic dataset.

Image illumination: Image illumination training wasperformed on a dataset containing 324 training images. Ofthese, 20% were images taken at night, 20% were imagestaken at dusk or sunset, 10% were images taken in snowy orfoggy conditions and the remaining 50% were images takenunder natural lighting conditions. Our illumination classi-fying algorithm was tested via 5-fold cross validation test-ing, and resulted in an accuracy of 78%. Since there is cur-rently no publicly available illumination benchmark datasetto compare against, we will make our training dataset pub-licly available for testing.

Image segmentation and labeling: Our imagesegmentation-labeling algorithm was tested on the Stanforddataset containing 715 images of outdoor scenes compiledby (Gould, Fulton, and Koller 2009); we performed thesame tests in the paper, a 5-fold cross-validation, with thedataset randomly split into 572 training and 143 test imagesfor each fold. Our pixel-level accuracy was 79.84% (0.77)

- standard deviation are given in parenthesis. A baselinestandard pixelwise CRF model gave a mean pixel accuracyof 74.30% (0.80) and the region-based energy model by(Gould, Fulton, and Koller 2009) had an accuracy of 76.40%(1.22). It is worth noting that their energy minimizationtechnique can take as long as ten minutes to segment oneimage, while our algorithm runs in only a few seconds forthe same image. Table 1 shows the confusion matrix for thelabeling on the same dataset.

A minor criticism we had of this dataset was its lack ofdiversity in scene categories. Out of the 715 images, morethan 50% were street views, while about 10% were imagesof a animals in an open field (many of which were obtainedfrom the MSRC dataset consisting mainly of a foregroundobject in the background). This dataset was more object-centric than general scene-based. Note:out of 715 images,there were only 14 images with the mountain label, henceits low diagonal score in Table 1.

Figure 5: Examples of images and our segmentation output.

Human-in-the-loop Evaluation: To test our sentencegeneration paradigm, we compiled a new dataset consistingof 709 images. The dataset included many more differentscene categories and objects than the Stanford dataset, al-though we “borrowed” several images from their dataset aswell as from Hoeim et al., (Hoiem, Efros, and Hebert 2007).About 75% of the images in the dataset are taken in daylightwhile the remaining images are equally distributed over hav-ing sunset (or dusk), foggy (or snowy) and night-time illumi-nation. The images with different scene illumination typeswere obtained from the Corel dataset as well as from googleimages.

Similarly, the distribution of scene categories is given asMountains - less than 5%, Woods - less than 10%, Beach(or coastal) scenes - about 25%, Street - about 20%, High-way - about 5%, Suburbs - about 5%, In-city - about 15%,Garden - less than 10%, Tall buildings about 10%, and Openplains - less than 10%. It is important to note that many im-ages strongly possess more than one scene category and aretherefore counted multiple times. Some examples of imagesfrom our dataset are shown in Figure 1 and Table 2. We thenused our DISCO paradigm, described in Sections 3 and 4 togenerate sentences for each image.

Our human-in-the-loop evaluation was conducted as fol-lows:• The evaluation begins with the user accessing http://www.zmldesign.com/SUnProject. This webpagerandomly selects one of the 709 images in the dataset andpresents its generated text to the user.

• The corresponding image along with 19 others (a total of20 images) are then presented to the user along with thegenerated text.

1491

Sky Tree Road Grass Water Buildings Mountain F object

Sky 90.58 1.37 0.40 0.00 0.09 6.09 0.14 1.32Tree 1.41 69.00 0.62 2.07 0.02 22.15 0.14 4.58Road 0.07 0.43 87.58 0.77 1.94 4.64 0.07 4.50Grass 0.10 7.05 5.24 80.81 0.01 3.93 0.022 2.62Water 4.74 0.33 24.34 0.18 59.11 4.63 1.28 5.39

Buildings 1.01 3.40 1.52 0.32 0.00 85.42 0.00 8.33Mountain 6.93 18.99 10.65 4.91 6.08 37.13 6.39 8.90F object 1.03 3.97 8.76 0.54 0.23 21.78 0.00 63.67

Table 1: Confusion matrix (corresponding to our recorded highest accuracy) from labeling the Stanford dataset

• The user is encouraged to view all 20 images in detailand select the image most appropriately described by thegiven text.

• A survey page is launched and the user is encouraged tofill the questionnaire. The questions on the survey include:(i) rating how well the description text explained the ac-tual image; and (ii) ranking the order of usefulness ofthe scene context, background regions, objects and peo-ple and color or shape in the presented sentences.

We conducted the web-based evaluation test with 25users. Each user was instructed to test about 10 times andthis was the case on average. We evaluated (i) the numberof times the users correctly selected the same image as wasbeing described; (ii) the user ratings of the description textcompared to the image and (iii) the rankings of the order ofusefulness of including different visual content informationin the generated sentences.

Because multiple images can have very similar text de-scriptions, the user is asked to rate the appropriateness andmeaningfulness of the sentence compared to expected se-lection. We provide this option so that even when 2 imagescorrespond to similar sentences, we are still able to mea-sure the meaningfulness of the sentences with respect to theexpected image. The user is also presented with a set ofquestions to determine which image aspect: scene catego-rization, background description, object detection or objectcolor and shape, yields the “best” information in the givenset of sentences. Figure 6 shows the distribution of how usersranked the sentence descriptions provided. With total scores

1: Sentence completely explains image2: Sentence satisfactorily explains image3. Sentence fairly explains the image4. Sentence poorly explains image5. Sentence does not explain image

Figure 6: Distribution of meaningfulness scores of generatedsentences

of Fairly explains (3) and higher, the algorithm was 60.34%meaningful. With scores of Satisfactorily explains (2) andhigher, the algorithm was 32.8% meaningful. 33% of se-lected images were exactly matched to the expected image,

although this is not conclusive evidence for sentence mean-ingfulness, since multiple images can have the same descrip-tions. Chance performance for Fairly explains (3) and higherwould be 15%, for Satisfactorily explains (2) and higherwould be 10%. If the test was run 250 times, chance per-formance of correctly selecting the right image each timewould be less than 1%.

Table 2 shows sample results from the different typesof image found in DISCO. Images tested include differentscenes at sunset, at night-time, in foggy or snowy weatherand in natural light. Several of the images shown are cor-rectly labeled and have correct sentences, while others haveincorrect labels, but with the sentences being quite general,are still descriptive and correct. Others yet have both incor-rect labels and sentences, and the descriptions are mislead-ing.

6 Discussion and Future Work

Sentences are extremely rich sources of information, bothfor transmitting and receiving information. In this paper,we have presented a sentence generation paradigm that isjudged by its meaningfulness to humans and 60.34% of thetests performed indicated that the generated sentences atleast fairly explained the presented images. Going forward,it will be useful to study what aspects of a scene yields themost information and under what circumstances. Our eval-uation test website provides the opportunity for the furtherevaluation tests. Also from our survey results, we intend tostudy the statistical distributions of the scene aspects thatusers found most useful and correlate these with the under-lying image content.

Also, by encouraging our testers to provide more appro-priate sentences for different images in the dataset, we arecollecting human annotated data that will be useful goingforward. We will also investigate how top-down models suchas presented by (Farhadi et al. 2010) will integrate with abottoms-up approach such as ours.

Acknowledgements

The authors would like to thank Randal Nelson for partic-ipating in the discussion leading to the development of thetesting paradigm. We would also like to thank the anony-mous reviewers for their insightful comments. IN is cur-rently funded by the NSF Computing Innovations (CI) Fel-lowship (2010), under the NSF subaward No. CIF-384.

1492

ReferencesAlpert, S.; Galun, M.; Basri, R.; and Brandt, A. 2007. Image seg-mentation by probabilistic bottom-up aggregation and cue integra-tion. In CVPR.Comaniciu, D., and Meer, P. 2002. Mean shift: A robust approachtoward feature space analysis. TPAMI 24(5):603–619.Farhadi, A.; Hejrati, S. M. M.; Sadeghi, M. A.; Young, P.;Rashtchian, C.; Hockenmaier, J.; and Forsyth, D. A. 2010. Everypicture tells a story: Generating sentences from images. In ECCV,15–29.Felzenszwalb, P. F., and Huttenlocher, D. P. 2004. Efficient graph-based image segmentation. IJCV 59:167–181.Felzenszwalb, P. F.; Girshick, R. B.; McAllester, D.; and Ramanan,D. 2010. Object detection with discriminatively trained part-basedmodels. TPAMI 32:1627–1645.Gould, S.; Fulton, R.; and Koller, D. 2009. Decomposing a sceneinto geometric and semantically consistent regions. In ICCV).Gupta, A.; Srinivasan, P.; Shi, J.; and Davis, L. 2009. Understand-ing videos, constructing plots learning a visually grounded story-line model from annotated videos. 2012–2019.Hoiem, D.; Efros, A. A.; and Hebert, M. 2007. Recovering SurfaceLayout from an Image. IJCV 75(1):151–172.Li, L.-J.; Socher, R.; and Li, F.-F. 2009. Towards total scene un-derstanding: Classification, annotation and segmentation in an au-tomatic framework. In CVPR, 2036–2043.Nock, R., and Nielsen, F. 2004. Statistical Region Merging.TPAMI. 26(11):1452–1458.Oliva, A., and Torralba, A. 2001. Modeling the Shape of theScene: A Holistic Representation of the Spatial Envelope. IJCV42(3):145–175.Rabinovich, A.; Belongie, S.; Lange, T.; and Buhmann, J. M. 2006.Model Order Selection and Cue Combination for Image Segmen-tation. In CVPR, 1130–1137.Reiter, E., and Dale, R. 2000. Building Natural Language Gener-ation Systems. New York, NY, USA: Cambridge University Press.Ren, X.; Fowlkes, C.; and Malik, J. 2005. Cue integration forfigure/ground labeling. In NIPS.Richard, X. H.; Zemel, R. S.; and perpi nan, M. A. C. 2004. Multi-scale conditional random fields for image labeling. In CVPR, 695–702.Shi, J., and Malik, J. 2000. Normalized cuts and image segmenta-tion. TPAMI. 22(8):888–905.Tu, Z.; Chen, X.; Yuille, A. L.; and Zhu, S.-C. 2005. Imageparsing: Unifying segmentation, detection, and recognition. IJCV63(2):113–140.Tversky, B., and Hemenway, K. 1983. Categories of environmentalscenes. Cognitive Psychology 49(15):121–149.Xiao, J.; Hays, J.; Ehinger, K.; Oliva, A.; and Torralba, A. 2010.SUN Database: Large Scale Scene Recognition from Abbey toZoo. In CVPR.Yao, Z.; Yang, X.; Lin, L.; Lee, M. W.; and Zhu, S. 2010. I2t:Image parsing to text description. Proc. of IEEE 98(8):1485–1508.

Actual image Regions labeled Text generated

The image scenery is set in the suburbs;There’s quite a large expanse of white sky inthe scenery. There’s quite a large expanse oftrees in the scenery. The wheat colored build-ings are prominent in the scenery.

The picture shows a scene... in the woods;The picture has a person in something white.The trees are quite prominent in the scenery.

The picture shows a scene... with a view oftall buildings; The picture background con-sists mainly of bluish-purple sky. The pic-ture background consists mainly of trees. Thebrown buildings are quite prominent in thescenery.

This scene is at sunset or dusk... with a viewof the water; There are at least 2 people in thepicture.

This scene shows a snowy day... in the woods;

The image scenery is set with a view of thewater; The picture has a person in some-thing brown. The person is standing on thewater. The green sky are quite prominent inthe scenery. The picture background consistsmainly of dark gray water. The dark graybuildings are quite prominent in the scenery.

The picture shows a scene... with mountainsin the background; The picture has a personin something dark gray. There is also a mo-torbike in the picture. There’s quite a largeexpanse of dark gray trees in the scenery

The picture shows a night scene... in a resi-dential neighborhood outside the city;

The picture shows a scene... in a street in thecity; There are at least 6 people in the pic-ture. Someone in the picture has on some-thing wheat colored. The people are standingon the road. The picture background consistsmainly of light gray buildings.

The image scenery is set in an open plainslandscape; There are at least 2 cars in the pic-ture including a purplish-pink one. The carsare on the road. There’s quite a large expanseof green trees in the scenery.

The image scenery is set with a view oftall buildings; The picture has a person insomething purplish-pink. The picture back-ground consists mainly of light gray build-ings. There’s quite a large expanse of darkgray trees in the scenery.

The picture shows a scene... in an open plainslandscape; There are at least 2 people in thepicture. Someone in the picture has on some-thing purplish-red. The people are standingon the sand. The dark gray buildings arequite prominent in the scenery. There’s quitea large expanse of dark gray trees in thescenery.

Table 2: Examples of images, labeled regions and generatedsentences.

1493