LETTER Communicated by Richard Baraniuk

Visual Categorization with Random Projection

Rosa I. Arriagaarriaga@cc.gatech.eduDavid Rutterquintopia@gmail.comGeorgia Institute of Technology, Atlanta, GA 30332, U.S.A.

Maya Cakmakmcakmak@cs.washington.eduUniversity of Washington, Seattle, WA 98195, U.S.A.

Santosh S. Vempalavempala@cc.gatech.eduGeorgia Institute of Technology, Atlanta, GA 30332, U.S.A.

Humans learn categories of complex objects quickly and from a few ex-amples. Random projection has been suggested as a means to learn andcategorize efficiently. We investigate how random projection affects cat-egorization by humans and by very simple neural networks on the samestimuli and categorization tasks, and how this relates to the robustness ofcategories. We find that (1) drastic reduction in stimulus complexity viarandom projection does not degrade performance in categorization tasksby either humans or simple neural networks, (2) human accuracy andneural network accuracy are remarkably correlated, even at the level ofindividual stimuli, and (3) the performance of both is strongly indicatedby a natural notion of category robustness.

1 Introduction

Humans learn to categorize stimuli into two or more sets and are able todo so accurately after being presented with a few training instances (Booth,2006; Makino & Jitsumori, 2007; Marcus, Vijayan, Rao, & Vishton, 1999;Mandler, 2003; Smith & Minda, 2000). This finding suggests that humanscan identify perceptual information in the stimuli that is relevant to rep-resenting a category and discriminating it from others. The vast amountof perceptual information that is continuously sensed by humans suggeststhat several filtering mechanisms operate on sensory information before itis used in higher-level cognitive processes such as planning, reasoning, orcategorization. Arriaga and Vempala (1999, 2006) suggested random pro-jection as a means by which the human brain can reduce the amount of

Neural Computation 27, 2132–2147 (2015) c© 2015 Massachusetts Institute of Technologydoi:10.1162/NECO_a_00769

Visual Categorization with Random Projection 2133

information that is processed when presented with stimuli. This letter ismotivated by the following question: Does random projection maintain es-sential properties of visual stimuli to allow for accurate categorization byhumans? In other words, does summarizing sensory data in this mannerhinder human performance? So far, this question has been considered onlytheoretically.

The human ability to categorize high-dimensional stimuli has been aninspiration for the categorization problem in machine learning. However,computational learning theory has frequently found learning in high di-mension to be an intractable problem (Valiant, 1984; Vapnik & Chervo-nenkis, 1971; Vapnik, 1995). To make the categorization problem moretractable, dimensionality reduction is used to focus on relevant informa-tion in high-dimensional data. Attribute selection methods try to assessthe relevance of dimensions in the high-dimensional space, so that onlythe most relevant subset of features is used for categorization. Projectionmethods try to find lower-dimensional spaces to represent data so that thecategorization problem is easier in the projection space.

Random projection (RP) is the method of mapping sample pointsin a high-dimensional space into a low-dimensional space whose co-ordinates are random linear combinations of coordinates in the high-dimensional space. An important property of RP, usually called the Johnson-Lindenstrauss lemma (Johnson & Lindenstrauss, 1984), is that distances be-tween pairs of points are approximately preserved by random projection,provided the target dimension is not too small (roughly the logarithm ofthe number of points being projected; see Vempala, 2004, for recent proofs).Arriaga and Vempala (1999, 2006) used this property to present a theoreticalmodel of efficient cognitive categorization. In their model, projected sam-ples from well-separated categories remain distinguishable after projection,and categorization can be achieved efficiently using the lower-dimensionalprojections of stimuli. In particular, they give a simple and efficient algo-rithm with near-optimal sample complexity for learning a large-marginlinear threshold function in a high-dimensional space: project data to a ran-dom low-dimensional space and then use a margin-based algorithm suchas perceptron; to label a new example, use the same projection and labelwith the learned threshold function. Their key insights are that marginsto well-defined class boundaries are preserved with high probability byrandom projection (for any single data point), and only a small number ofpoints are needed for training in the low-dimensional space; therefore, withlarge probability, a sufficiently large sample maintains a large margin in theprojected space. They show that the margin of separation of several math-ematically well-defined concept classes is preserved by random projection,so that learning the concept is possible and efficient in the projected sub-space. Moreover, random projection is easily realized by a simple two-layerneural network with edge weights set independently and randomly. In fact,setting each weight randomly to −1 or 1 suffices, as shown by Arriaga and

2134 R. Arriaga, D. Rutter, M. Cakmak, and S. Vempala

Vempala (1999, 2006), who called it neuron-friendly random projection. Re-cent work (Allen-Zhu, Gelashvili, Micali, & Shavit, 2014) shows that theweights out of each node can all have the same sign and still enjoy dis-tance (and margin) preservation. The output of a single neuron is believedto have the same sign on all synapses (excitatory or inhibitory), therebymaking random projection even more plausible.

Random projection has been widely applied in conjunction with othermethods in machine learning, such as manifold learning (Hegde, Daven-port, Wakin, & Baraniuk, 2007; Hegde, Wakin, & Baraniuk, 2007; Freund,Dasgupta, Kabra, & Verma, 2007), face recognition (Goel, Bebis, & Nefian,2005), mixture learning (Dasgupta, 1999, 2000), and concept learning (Ar-riaga & Vempala, 2006; Garg & Roth, 2003). Bingham and Mannila (2001)and Goel et al. (2005) show that random projection has comparable perfor-mance with conventional dimensionality-reduction methods such as prin-cipal component analysis while significantly reducing the computationalcost and being data independent. RP has been proposed as an alternativeto kernel methods (Balcan, Blum, & Vempala, 2006). It has also been foundto be useful in training deep neural networks (Saxe et al., 2011) as either aseparate layer or a good initialization.

In this letter, we study random projection in the context of human cog-nition. We build on the idea that visual perception may involve a randomprojection stage. Random projection complements existing mathematicalframeworks for cognition (Marcus, Vijayan, Rao, & Vishton, 1999; Tenen-baum, Kemp, Griffiths, & Goodman, 2011; Xu & Kushnir, 2013) and can beviewed as a general-purpose preprocessing step for human learning. Wemake predictions based on this hypothesis and test these predictions usingbehavioral experiments with human subjects as well as simple neural net-works designed to capture predicted human behavior. The main idea is thathuman accuracy at categorization tasks should not degrade if stimuli arerandomly projected in advance. Our first hypothesis (H1) is the following(in section 4, we place it in the context of visual psychophysics):

H1: Humans should be able to learn to categorize randomly projectedstimuli as easily and accurately as they categorize unprojectedstimuli.

Our second hypothesis is based on the idea that predicted human be-havior on perceptual categorization tasks will be mimicked by very simpleneural networks under the same stimulus conditions. The use of neuralnetworks in machine learning is a sophisticated and highly successful tech-nique today, but our goal is not to find a neural network with the bestpossible performance on the categorization tasks at hand. Rather, we usethe simplest neural networks in order to draw robust and meaningful con-clusions. Moreover, all of our categorization tasks are based on one-shottraining—those with only a single presentation of one example from eachof two categories. While there has been work on Bayesian learning with

Visual Categorization with Random Projection 2135

one-shot training (Li, Fergus, & Perona, 2006; Lake, Salakhutdinov, & Tenen-baum, 2013), multilayer neural networks typically need a large number ofexamples to generalize well and achieve low classification error (LeCun &Bengio, 1998; Hinton et al., 2012). In light of this, we hypothesize:

H2: Very simple neural networks modeling categorization with randomprojection should achieve similar performance as humans.

Our experimental design to test these predictions is presented in the nextsection. One challenge in the design is to create stimuli and categories thatare both natural and novel (so as to avoid the bias of prior knowledge ofcategories while not ignoring natural features). We developed three typesof stimuli and used two types of random projection.

In section 3, we present the results of our human categorization experi-ments on original and projected visual stimuli. They provide strong supportfor our hypotheses in each of the stimuli types and each of the projectiontypes. We also found similar accuracy in experiments with humans andsimple neural networks and observed improved efficiency during catego-rization when done with random projection. These results are discussedin detail in section 3. We then turn to a more detailed analysis, comparingthe performance of humans and neural networks on individual stimuli. Wefound that these are highly correlated, with high agreement in the subsets ofstimuli that humans and neural networks categorized incorrectly. To under-stand this, we introduce a natural notion of robustness of stimuli. Roughlyspeaking, it measures how clearly a stimulus belongs to one category versusthe other, with the robustness value being closer to zero as a stimulus be-comes more ambiguous. Placing the images in the order of robustness, wefind both that the performance of humans and neural networks improveswith stimulus robustness and that the few stimuli where humans and neu-ral nets disagree are almost all in the region of low robustness. These resultsare presented in section 3.1.

Although we cannot definitively claim that the human brain actuallyengages in random projections, our results support the notion that randomprojection is a plausible explanation for dimensionality reduction withinhuman perception via random summarization. To the best of our knowl-edge, this is the first study of random projection based on human subjects.

2 Methods

To address the question of how random projections affect categorizationability in humans, we conducted experiments in which human subjectswere tested on two-class visual categorization tasks with both unprojectedand projected visual stimuli. We designed three types of visual images andtwo types of random projection methods. We also designed two versions of asimple neural network to categorize the same visual stimuli. In this section,

2136 R. Arriaga, D. Rutter, M. Cakmak, and S. Vempala

Figure 1: Samples from the two categories A and B for the three types of stimuli.

we provide details about the visual stimuli, the projection methods, theexperimental design, and the neural network design.

2.1. Visual Stimulus Design. Our choices in creating these stimuli stemfrom two criteria, naturalness and novelty. With naturalness, we tried to en-sure that the stimuli reflect properties of the real world. In particular, theyshould have a clear structure, with different connected regions distinguish-able by differences in color, an attribute that the visual system pays distinctattention to (Read, 2014); they should not be collections of arbitrarily col-ored pixels. On the other hand, we did not want the images to appearsimilar to categories that subjects might already have knowledge of or havereal-world experience with. Using images of dogs and cats would have in-evitably caused the subjects to call on prior knowledge of these categories.

We created three types of images to use as visual stimuli: Geon arrays,which are arrays of simple geometric shapes; Dartboards, which are coloredVoronoi diagrams; and substrates which are simple colored layers rotated bysome amount. Samples of the two categories from each type of visual stimuliare given in Figure 1. The images for each category were generated by thesame algorithm using a different set of parameters for the two classes. Ingeon arrays, we used four shapes (geons): balls, cones, cubes and cylinders.Images were generated by placing nine random geons in a 3 × 3 array,with different distributions on geons for the two categories (A: 0.15, 0.2,0.05, 0.6; B: 0.2, 0.6, 0.1, 0.1). Dartboard images were generated by pickingfour points at random in a unit square, computing their Voronoi diagram,and coloring each region; again, different distributions on the locations ofthe points were used for the two categories. For category A, a point waschosen in the upper half of the main diagonal of the containing square,giving a partition into four rectangles via the horizontal and vertical linesthrough the chosen point; then four points were picked randomly, onefrom each rectangle. For B, the point used for subdivision was chosenfrom the lower half of the diagonal. Substrates were generated using one-dimensional Brownian random walks from left to right to partition intolayers; then the layers were colored and rotated randomly. The angle of

Visual Categorization with Random Projection 2137

Figure 2: Description of the random projection methods for visual stimuli:(a) sliding window random projection and (b) corner random projection.

rotation was from a different distribution for the two categories (a uniformlychosen anticlockwise rotation for A and clockwise for B).

These categories can be assigned semantic labels, at least beforeprojection—for example, “more cylinders” versus “more cones,” “moregreen” versus “more white,” and “sloping up” versus “sloping down.”However, these are not familiar categories; rather, they are familiar features(number, color, angle) in an unfamiliar context. In our experiments, sub-jects were presented with a single pair of images from each stimuli type,one from each category, for a few seconds. Immediately following this, theyhad to classify other images, one at a time; the labels of images used fortesting were not revealed to the subject. Thus, while subjects must haveused some cognitive rule to categorize, it is unlikely that they had the timeto consciously define the right semantic categories and then use them forclassification; this is even more so in the case of projected stimuli, which wedescribe next.

2.2. Random Projection of Images. Random projection maps points inn-dimensional space to k-dimensional space, where k is smaller than n. Theprocess of projection can be achieved by multiplying a given point (taken asan n-vector) by a random n × k matrix to produce a k-vector. Typical choicesfor the random matrix with provable guarantees are picking each entryindependently from a standard normal distribution (Johnson & Linden-strauss, 1984; Indyk & Motwani, 1998; Arriaga & Vempala, 1999; Dasgupta& Gupta, 2003), or uniformly from two discrete values {−1, 1} (Arriaga &Vempala, 1999, 2006). In designing our projections, we tried to ensure thatthe projected stimuli have a visually salient representation so that com-parison with the original stimuli is meaningful. This suggests that somespatial properties of the image should be preserved by projection and thatprojections have meaningful color values. We used two projection methods(see Figure 2), as well as unprojected images. Nevertheless the projectedimages were only of size 6 × 6, and the projection methods were genericand applied to all three stimulus types without any modification.

2138 R. Arriaga, D. Rutter, M. Cakmak, and S. Vempala

Figure 3: Samples of images projected using sliding window and corner pro-jection methods.

In the sliding-window random projection, a single color is generatedfrom a random 0-1 combination of the colors in a window that slides overthe image and that color is used to fill a corresponding location in the pro-jected image. The window size and the increment with which the windowis moved determine the reduced image dimension. This is a random con-volution of a particularly simple type, with random weights chosen from{0, 1} and a very small number of units. Neural networks based on con-volution have been extremely successful in large-scale machine learning(LeCun & Bengio, 1998; Sermanet et al., 2014). They are usually initializedwith random weights and then trained with a large number of examples.Biological support for such mechanisms indicates that subsets of sensoryinputs are aggregated at lower levels (e.g., line detectors for various parts ofthe visual field). Our choice of simplified convolution is considerably morerestricted than general random projection, and thus any support for our hy-pothesis (of no degradation in performance) would arguably be stronger.In our experiments, we used a window to reduce the resolution of originalimages from 150 × 150 to 6 × 6 (for the geon arrays) and 500 × 500 to 6 × 6(for dartboards and substrates). In the former case, the window size was25 × 25, and in the latter it was 100 × 100.

Another aspect of the human visual system is the existence of specializedfeature detectors for colors, lines, corners, and so on. We designed a randomprojection that takes this into account by first identifying corners in an im-age. In corner random projection, corners in the original image are detectedusing the FAST corner detector (Rosten & Drummond, 2006) and windowsof pixels are extracted from a small region, 31 × 31 around each corner. Theprojections are computed as the weighted average of these windows usinga set of randomly chosen weights (see Figure 2). In this case, the resolutionreduces from 150 × 150 and 500 × 500 to 20 × 20.

Samples of projected images obtained with both methods are given inFigure 3. These families of random projections do not have distance (ormargin) preservation guarantees for arbitrary inputs. Our prediction is thathuman performance on categorizing these visual stimuli will not be affectedin any significant way by these projections.

Visual Categorization with Random Projection 2139

2.3. Procedure. There were nine experimental conditions resulting fromall combinations of the three stimuli types (geon arrays, dartboards, andsubstrates) and the three projections (unprojected, sliding window pro-jected, and corner projection). Each of the nine conditions was tested on 16subjects. Subjects were randomly assigned to three conditions each, cho-sen to range across the three stimuli types and the three projections andbalanced across them. For example, a subject might be tested on unpro-jected geons, sliding window–projected dartboards, and corner-projectedsubstrates. The total number of human subjects was thus 48 (= 9 × 16/3).All the subjects were college students ages 18 to 24, with an equal numberof males and females. They signed an IRB-approved consent form beforeparticiipating in the study

In each categorization task, subjects were first shown two sample images,one from each category, side by side, for up to 10 seconds (the trainingphase). Then each subject was shown a sequence of 16 single images ofthe same type of stimulus and asked to label each one as A or B within 5seconds. Images from the two categories were presented in random order.The number of correct responses and the reaction time were measured.

2.4. Neural Network Categorization Task. For the categorization ofvisual stimuli into two classes, we used two simple feedforward neuralnetwork structures. The first one assumes that all images (original or pro-jected) are scaled to a 150 × 150 resolution. The input to the neural networkis the visual stimuli with RGB values normalized, and the sign of the outputis the category label. The neural networks contain 150 × 150 × 3 = 77, 500input units and 1 output unit. The input units are directly connected tothe output unit, and all transfer functions used are linear. Essentially thecategory is decided by the sign of the inner product of the input vectorand the weights—a linear threshold function with zero as the threshold.The weights are initialized randomly in the range [−1, 1], and the vectorof weights is normalized. The weights are updated with the delta rule, andthe neural network is trained until the total weight update within an epochfalls below a certain threshold.

The second neural network differs from the first in that projected imagesare used as inputs without being rescaled. Therefore, the neural networksare much smaller for projected stimuli. The complete neural network is thecombination of a projection layer, in which no learning occurs (the weightsare chosen randomly and fixed), and a categorization layer. The input tothese networks is therefore always the unprojected stimuli. The two neuralnetworks are illustrated in Figure 4.

To mirror the task performed by humans, we trained the neural networkson only one image pair (one from each category) and tested them on theremaining 16 images for that task, identical to the human experiment. First,we trained using the exact same exemplars that the humans were presented.Later, to obtain an average performance of the neural network, we did a

2140 R. Arriaga, D. Rutter, M. Cakmak, and S. Vempala

Figure 4: The simple neural network used for visual stimuli categorization.

nine-fold cross-validation, whereby the neural network is trained with asingle stimulus pair and tested on the remaining images. Every stimulus inthe set is used once as a training example, for a total of 9 iterations.

3 Results

In this section, we present comparative results of categorization perfor-mance for unprojected and projected images for each type of stimulus. Wecompare human subjects and the simple neural networks used, and weanalyze both average performance and performance on individual images.

In Figure 5a, the average successful categorization rate among humansubjects is compared for unprojected and projected images. Humans wereable to categorize successfully in all three types of images (t-test withp < 0.001). Our results indicate that in spite of the reduction in the fea-ture complexity, there was no statistically significant loss in subjects’ abilityto correctly categorize stimuli. Thus, the random projections used in ourexperiments preserved the essential similarities and differences betweenstimuli from different categories as perceived by humans. Also, after pro-jection, the accuracy of classification was similar across all three stimulitypes and both projection methods. Moreover, posttest responses to thequestion, “What did you use to categorize?” were highly varied, exceptfor unprojected substrates, suggesting that human subjects were not con-sciously creating the same semantic labels for categories.

Similar results were obtained in the neural network categorization task.Here, for the projected images, we used two types of networks as describedin the previous section (see Figure 4; the first one uses images rescaled to150 × 150, regardless of projection, and the second uses a fixed projectionlayer and does not rescale). As expected, there was little difference in theperformance of the two types of neural networks. The networks with andwithout the rescaling had identical performance in six of the nine con-ditions, and the difference in the classification rate was just under 7% inthe remaining three conditions (unprojected and sliding window–projected

Visual Categorization with Random Projection 2141

Figure 5: Experimental results. The fraction of correctly categorized stimuliacross all stimulus types without projection (blue), after sliding window ran-dom projection (red), and after corner projection (yellow) by (a) humans and(b) neural networks through the nine-fold cross validation are shown. Panelsc and d give the performance for each stimulus type. Standard deviations aregiven by the error bars.

geon arrays and corner-projected substrates). Projections improved the effi-ciency in categorization by reducing the size of the neural network. Figure 5includes neural network results without the projection layer. Figures 5a and5b compare performance on original and projected stimuli for humans andneural networks. The neural network results are the average performancein the nine-fold tests. Figures 5c and 5d compare performance across the dif-ferent types of stimuli. Our findings were similar across all stimulus types,indicating that the general-purpose random projection methods used didnot affect human or neural network performance.

We extended our analysis by investigating variations in successful cate-gorization on individual images.

2142 R. Arriaga, D. Rutter, M. Cakmak, and S. Vempala

3.1. Comparison Based on Robustness. Here we investigate how cat-egorization of images relates to their robustness, which we will define.Although category robustness has been discussed in the cognitive scienceliterature in relation to categorization, a concise mathematical definitionof robustness is not available. Arriaga and Vempala (1999) define the ro-bustness of a category as the minimum, over all examples, of the distanceof an example to the category boundary. This assumes that the categoriesare defined by concepts with well-defined boundaries, while in our case,no assumption about the category representation is made. The linear dis-criminant analysis method, also known as Fisher’s method (Fisher, 1938),estimates the hyperplane that best separates two categories by maximizingthe ratio of the distance between categories to the scatter within categories,after approximating categories by normal distributions. We adopt the ra-tio between the distance of the sample to the mean of its category to thedistance of the sample to the mean of the other category. In order to unifythe robustness calculation for the two categories, we take the difference be-tween the ratio and its reciprocal. The precise definition of our robustnessmeasure is as follows. Here x is the sample stimuli and μA and μB are themeans of the two categories:

RAB(x) = ‖x − μA‖‖x − μB‖ − ‖x − μB‖

‖x − μA‖ .

Note that this provides a measure of robustness for individual samplesrather than for a whole category. For an example whose category isambiguous—its distances to the two sample means are nearly equal—therobustness measure will be close to zero. If the distance to one of the cat-egories is smaller by a factor of r, the robustness measure will go up asr − 1/r in magnitude (positive for examples from one category and nega-tive for examples from the other category).

We investigated the role of robustness in categorization by looking at thecategorization performance of humans and neural networks on individualimages, taking into account their robustness. We first observed that sampleson which human subjects made the most mistakes are the ones that arefrequently misclassified by the neural network. In Figure 6, performances onindividual images of human subjects and neural networks (in the nine-foldcross-validation) are compared. Although there exist outliers, performancein general is very similar, even on individual stimuli.

We argue that the performance on individual images is strongly relatedto the robustness of the image as defined above. In Figure 6, stimuli aresorted according to their robustness, the two extremes being the most robustsamples of the two categories and the middle ones being the least robustones. It is again clear from this figure that images with low robustnessare the ones on which both humans and neural networks perform poorly.

Visual Categorization with Random Projection 2143

Figure 6: (Left) Performance comparison on individual image samples (geons)sorted according to robustness. (Right) Distribution of image samples accordingto their robustness and change of human (top) and neural network (bottom)performance with robustness.

2144 R. Arriaga, D. Rutter, M. Cakmak, and S. Vempala

Another interesting observation is that most samples on which human andneural network performance are not correlated (outliers) are ones with lowrobustness.

The distribution of image samples on a robustness scale in Figure 6shows that the trend of degrading performance toward the origin can beobserved in all categories. The figure also provides an idea of how projec-tions affect robustness. While sliding window projection does not seem tochange the robustness of samples, corner projection stretches the robust-ness distribution, making low-robustness images worse and highly robustimages better.

4 Discussion

The main findings of this study can be summarized as follows. First, thecategorization ability of human subjects does not degrade with randomprojection applied as a task-independent initial processing step. Second,simple neural networks have remarkably similar performance to humanson categorization tasks for both original and projected stimuli. Despite theirsimplicity, these networks are able to capture human categorization capa-bilities at the level of individual stimuli. And third, human and neuralnetwork performance are highly correlated to the robustness of categories.A measure of robustness that is in agreement with human performancecan be obtained by considering both the distance of samples to the othercategory and the distance to the prototype of their own categories. Thereexists a three-way correlation between robustness of image samples andthe categorization performance by humans and neural networks on thoseimages. As far as we know, this is the first study using random projec-tions with human subjects and comparing their performance on one-shotlearning and categorization with neural networks.

Turning to support from psychophysics (the analysis of perceptual pro-cess using quantitative measures), we note that it is possible to comparemonkey neuronal data to human psychophysics performance because thetwo are generally understood to have very similar visual systems. Resultsindicate that humans and nonhuman primates can process visual stimulirapidly, at speeds between 14 and 50 ms (Proctor & Brosnan, 2013). Thishappens under a variety of presentation paradigms from single stimuluspresentation of clip art to a continuous sequence of naturalistic images(Keysers, Xiao, Fldik, & Perrett, 2001). The pyschophysics literature alsoshows consistency of encoding among naturalistic and geometric stimuliin monkey primary visual cortex (Wiener, Oram, Liu, & Richmond, 2001).Our results raise a number of questions about how the visual system mightbe able to make random projections of stimuli. The literature indicates thatif RP is a mechanism in the visual system, then it (1) it happens quickly, (2)in both human and nonhuman primates, and (3) across varied stimuli thatare static and in motion.

Visual Categorization with Random Projection 2145

To conclude, random projection is computationally efficient and couldaccount for the efficiency of categorization by humans. In neural networks,random projection increases the efficiency of learning by reducing the sizeof the network. Taken together with our empirical findings that preproject-ing visual stimuli does not significantly affect human ability to categorize,random projection appears to be a highly plausible mechanism for infor-mation processing in general and categorization in particular. Confirmingthis, by understanding the projections used more precisely, is a compellingresearch direction for cognition and learning. As one reviewer suggested, itwould be interesting to compare human and neural network performanceon more complex images, with more training examples and using deeperneural networks. Future studies could develop stimuli by varying the easewith which they can be labeled semantically (easy to hard) and vary therandom projection method from what we used and other neurally plausi-ble methods to fully random positive and negative weights. In this regard,we are inspired by the fact that the trichromatic nature of human visionwas deduced through behavioral experiments hundreds of years beforeobjective knowledge from physiology confirmed its existence (Read, 2014).

Acknowledgments

We thank David Mann, Dawn Finney, Ya-Lin Huang, and Parikshit Ram fortheir help at various stages of this project. We also thank the anonymousreviewers for many useful comments that improved the letter.

References

Allen-Zhu, Z., Gelashvili, R., Micali, S., & Shavit, N. (2014). Sparse sign-consistentJohnsonlindenstrauss matrices: Compression with neuroscience-based con-straints. Proceedings of the National Academy of Sciences, 111(47), 16872–16876.

Arriaga, R., & Vempala, S. (1999). An algorithmic theory of learning: Robust conceptsand random projection. In Symposium on the Foundations of Computer Science (pp.616–623). Piscataway, NJ: IEEE.

Arriaga, R., & Vempala, S. (2006). An algorithmic theory of learning: Robust conceptsand random projection. Machine Learning, 63(2), 161–182.

Balcan, M., Blum, A., & Vempala, S. (2006). Kernels as features: On kernels, margins,and low-dimensional mappings. Machine Learning, 65(1), 79–94.

Bingham, E., & Mannila, H. (2001). Random projection in dimensionality reduction:Applications to image and text data. In Proceedings of the Seventh ACM SIGKDDInternational Conference on Knowledge Discovery and Data Mining (pp. 245–250).New York: ACM.

Booth, A. (2006). Object function and categorization in infancy: Two mechanisms offacilitation. Infancy, 10(2), 145–169.

Dasgupta, S. (1999). Learning mixtures of gaussians. In Symposium on the Foundationsof Computer Science (pp. 634–644). Piscataway, NJ: IEEE.

2146 R. Arriaga, D. Rutter, M. Cakmak, and S. Vempala

Dasgupta, S. (2000). Experiments with random projection. In Proceedings of the Six-teenth Conference on Uncertainty in Artificial Intelligence (pp. 143–151). San Fran-cisco: Morgan Kaufmann.

Dasgupta, S., & Gupta, A. (2003). An elementary proof of a theorem of Johnson andLindenstrauss. Random Structures and Algorithms, 22(1), 60–65.

Fisher, R. (1938). The use of multiple measurements in taxonomic problems. Annalsof Eugenics, 7, 179–188.

Freund, Y., Dasgupta, S., Kabra, M., & Verma, N. (2007). Learning the structure ofmanifolds using random projections. In J. C. Platt, D. Koller, Y. Singer, & S. T.Roweis (Eds.), Neural information processing systems, 20. Cambridge, MA: MITPress.

Garg, A., & Roth, D. (2003). Margin distribution and learning. In Proceedings of theInternational Conference on Machine Learning (pp. 210–217). Cambridge, MA: MITPress.

Goel, N., Bebis, G., & Nefian, A. (2005). Face recognition experiments with randomprojection. In Proceedings of SPIE: Biometric Technology for Human Identification II(pp. 426–437). Bellingham, WA: SPIE.

Hegde, C., Davenport, M., Wakin, M., & Baraniuk, R. (2007). Efficient machine learn-ing using random projections. In Workshop on efficient machine learning, neuralinformation processing systems. Cambridge, MA: MIT Press.

Hegde, C., Wakin, M., & Baraniuk, R. (2007). Random projections for manifoldlearning. In J. C. Platt, D. Koller, Y. Singer, & S. T. Roweis (Eds.), Advances inneural information processing systems, 20. Cambridge, MA: MIT Press.

Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., . . . Kingsbury,B. (2012). Deep neural networks for acoustic modeling in speech recognition: Theshared views of four research groups. IEEE Signal Processing Magazine, 29, 82–97.

Indyk, P., & Motwani, R. (1998). Approximate nearest neighbors: Towards remov-ing the curse of dimensionality. In Proceedings of the Symposium on the Theory ofComputing. New York: ACM.

Johnson, W., & Lindenstrauss, J. (1984). Extensions of Lipshitz mapping into Hilbertspace. Contemporary Mathematics, 26, 189–206.

Keysers, C., Xiao, D.-K., Fldik, P., & Perrett, D. I. (2001). The speed of sight. Journalof Cognitive Neuroscience, 13, 90–101.

Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2013). One-shot learning by in-verting a compositional causal process. In C. J. C. Burges, L. Bottou, M. Welling,Z. Ghahramani, & K. Q. Weinberger (Eds.), Advances in neural information process-ing systems, (pp. 2526–2534). Red Hook, NJ: Curran.

LeCun, Y., & Bengio, Y. (1998). Convolutional networks for images, speech, andtime series. In M. Arbib (Ed.), The handbook of brain theory and neural networks.(pp. 255–258). Cambridge, MA: MIT Press.

Li, F., Fergus, R., & Perona, P. (2006). One-shot learning of object categories. IEEETrans. Pattern Anal. Mach. Intell., 28(4), 594–611.

Makino, H., & Jitsumori, M. (2007). Discrimination of artificial categories structuredby family resemblances: A comparative study in people (Homo sapiens) and pi-geons (Columba livia). Journal of Comparative Psychology, 121(1), 22–33.

Mandler, J. (2003). Early category and concept development: Making sense of theblooming, buzzing confusion. In D. Rakinson & L. Oakes (Eds.), Conceptual cate-gorization. New York: Oxford University Press.

Visual Categorization with Random Projection 2147

Marcus, G., Vijayan, S., Rao, S. B., & Vishton, P. (1999). Rule-learning in seven-month-old infants. Science, 283, 77–80.

Proctor, D., & Brosnan, S. (2013). Visual processing speed in capuchin monkeys(Cebus apella) and rhesus macaques (Macaca mulatta). International Journal of Com-parative Psychology, 26, 166–175.

Read, J. (2014). The place of human psychophysics in modern neuroscience. Neuro-science, 296, 116–129.

Rosten, E., & Drummond, T. (2006). Machine learning for high-speed corner detec-tion. In European Conference on Computer Vision (Vol. 1, pp. 430–443). New York:Springer.

Saxe, A. M., Koh, P. W., Chen, Z., Bhand, M., Suresh, B., & Ng, A. Y. (2011). On randomweights and unsupervised feature learning. In Proceedings of the 28th InternationalConference on Machine Learning (pp. 1089–1096). Madison, WI: Omnipress.

Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2014).Overfeat: Integrated recognition, localization and detection using convolutionalnetworks. In International Conference on Learning Representations. N. P.: Computa-tional and Biological Learning Society.

Smith, D., & Minda, J. (2000). Thirty categorization results in search of a model.Journal of Experimental Psychology: Learning, Memory, and Cognition, 26(1), 3–27.

Tenenbaum, J. B., Kemp, C., Griffiths, T. L., & Goodman, N. D. (201). How to growa mind: Statistics, structure, and abstraction. Science, 331(6022), 1279–1285.

Valiant, L. (1984). A theory of the learnable. Communications of the ACM, 27(11),1134–1142.

Vapnik, V. (1995). The nature of statistical learning theory. New York: Springer.Vapnik, V., & Chervonenkis, A. (1971). On the uniform convergence of relative

frequencies of events to their probabilities. Theory of Probability and Its Applications,26(2), 264–280.

Vempala, S. (2004). The random projection method. Providence, RI: American Mathe-matical Society.

Wiener, M., Oram, M. W., Liu, Z., & Richmond, B. (2001). Consistency of encodingin monkey visual cortex. Journal of Neuroscience, 21, 8210–8221.

Xu, F., & Kushnir, T. (2013). Infants are rational constructivist learners. Current Di-rections in Psychological Science, 22(1), 28–32.

Received December 9, 2014; accepted May 27, 2015.