GiveWell August 18 2011 Research Event Transcript

7/31/2019 GiveWell August 18 2011 Research Event Transcript

1/40

INTRODUCTIONS AND STAFF INTRODUCTIONS [0:00:01-0:02:49]

Elie: [0:00:01]I just wanted to first thank everyone for coming, were reallyexcited that you all could make it.

[0:00:07] One, a couple of notes; first, I said this in the email I sent outyesterday, or two days ago, that were planning to record this meeting, kindof in line with how GiveWell generally operates. If theres anything thatyou end up saying that youd prefer we just dont put in the recording wecan cut it out, so just let us know afterwards and were happy to do that.

Man 1: Or you can just say, Off the record, before you say it.

Elie: [0:00:29] Yep. But before we get started I just wanted to quickly have theGiveWell staff introduce themselves. So, Im Elie Hassenfeld, Im one ofthe cofounders of GiveWell.

Holden: Im Holden Karnofsky, Im the other cofounder.

Natalie: Im Natalie Stone and Ive been an employee for about two years now.

Wendy Knight: Im Wendy Knight and I [inaudible 0:00:49]

Alexander: Alexander Berger, Ive been on for about six weeks [inaudible 0:00:57]

Josh: Im Josh Rosenberg and Im a [inaudible 0:00:58]

Stephanie: Im Stephanie Wykstra. I came from [inaudible 0:01:05] and Ive been herefor about three months now.

Elie: [0:01:09] So our goal with tonight is to try to have an open forum aboutGivewells research. One of our major goals at the GiveWell project is tocreate an open conversation about how to give and what givingopportunities are going to be most effective. We know that we put out a lotof material on our website, we publish a lot of information and it cansometimes be difficult to engage with just the written material so we hopethat tonight may offer people another type of opportunity to offer feedbackand give us thoughts on our research and how were going about what weredoing. This is the first time that were holding this and if its successful wehope to hold more events like this in the future.

[0:01:51] Just a quick overview of our plan for how were going to go aboutthis tonight, were just going to do a few brief presentations. Holden isgoing to start off and give a background on our research process and ourresearch progress so far this year. Itll be quick and well pause forquestions after that, as we will pause after each of the little presentations


2/40

that we make. Then Im going to talk about some of the mostinterestingorganizations that weve looked at this year, both the ones thatwe think are most likely to receive our highest rating at the end of the yearand also some organizations that we felt like our investigation led us tolearn a lot about what they do and whether it works. And then finally,

Holden is going to talk a little bit more about the future of GiveWell.

[0:02:33]Throughout we hope that you, we see this as an opportunity foryou to ask any questions about what we say, but also any burning questionsyouve always had about GiveWell, but never had the opportunity to ask.So we hope that you take the opportunity and engage with us in what weredoing.


3/40

BACKGROUND ON RESEARCH PROCESS [0:02:50-0:11:07]

Holden : [0:02:50] So, Im Holden and Im going to talk about a quick backgroundand also our research process. Im going to keep the background part reallyshort because people here are pretty much insiders, theyre pretty much

GiveWell fans. Were starting to grow with this kind of event and in thefuture we might be [doing this thing more widely but thats where we arenow. But Im not giving a lot of background, the main thing that I want tosay at this point is that GiveWell has been working fulltime for a little overfour years to find the best giving opportunities you can. In some ways thatsa while, but given what were trying to do and the complexity of it, fouryears is not really very much in my opinion. I want to be clear to everyonethat we dont think that were experts, we dont think weve figured it allout, so we hope that everyone will keep that in mind.

[0:03:41] Were going to be totally open, were going to say whats on our

minds. We would be happy if everyone else in this room did the same thing,dont worry about contradicting us, dont worry about challenging us, welike that, and dont worry about sounding nave because I think that mostpeople are probably nave about the kinds of problems that were trying tosolve and would assume that theyre trying to do the best they can.

[0:04:01] So with that in mind Im just going to talk about what our goal isright now with our research and how were trying to accomplish that goaland were going to pause a have a discussion with that.

[0:04:11] Basically GiveWell evolved a little bit over the years and rightnow our top priority is defined by what I might call the next VillageReachby which I mean the next charity that were really excited to raise as muchmoney for as we can. This is maybe not exactly the same as what we set outto do in past.

[0:04:41] Were trying to find charities we could really get behind, so itsnot, you know, in the past I think we may have been putting more effortinto inspect charities [inaudible 0:04:50] different causes, rate as manycharities as we can, find charities that meet certain predefined metrics.Thats really what we realized is the most important thing about GiveWellis that were putting out a number one charity that we feel really good aboutand getting as much money to it as we can. [inaudible 0:05:08] more ratingsto do with research.

[0:05:14]So for that end weve broadened our scope. We used to look onlyfor things that were what we called prudent, cost effective, and scalable.There were a lot of types of charities we just wouldnt consider, thatschanging, somewhat gradually, but its is changing. So now were open to awhole bunch of different types of charities and particularly charities that


4/40

may have a low probability of success, but a very high value if theysucceed, something that we hadnt dealt with much in the past, but weremuch more interested in now. As a result of doing that its required somechanges to our process because things are much less [inaudible 0:05:48]now, were looking at lots of different types of organizations, that were

trying to be more open ended about how we approach them.

[0:05:54] So the basic process we have is just; Step 1, look at as manycharities as we can, consider as many as we can. People ask, How do youchoose which charities to examine? Well, its every charity that we canverify the existence of, basically, within the category of developing worldaid. We look at all of them and generally based on either things that weheard because we do try to talk to people and ask around or based on thewebsite of the charity, well flag them as promising for a variety of differentreasons. A couple of the old reasons that we had for flagging a charitywould be that they were working on a priority program or that they were

publishing some evidence of effectiveness, now we have a whole bunch ofother things and theyre in the packetthat we handed out, like having somekind of prior [inaudible 0:06:41] ideas, part of that being recommended byfunders that we think we can put to weight on, its part of that.

[0:06:47] Once we have a list of flagged charities, what I mean by that isjust charities that we think stand out in some very preliminary way, thatswhen well start figuring out which order to contact them in and thatsreally just based on subjective determination of which one we think is goingbe the outstanding charity which were looking for. Earlier this year weprobably contacted about 10 charities and were trying to ramp things up abit and know where we stand on them before we contacted more. A littlewhile ago we stepped back and said, This is not a good use of time. Onceweve gone back and forth to the charity and were not as excited about itwe need to put that time into contacting more. So weve accelerated thepace of contacting new charities and move over the amount of work wereputting in to closing the whole deal and figure whats going to get publishedand all that. Also, in addition, we have two temporary employees thissummer. Our staff has grown lately so our pace of contacting the charitieshas accelerated a lot and now were really opening conversations with a lotof charities.

[0:07:51] So the first step when we decide to contact the charity is just avery open ended conversation. These used to be hard to get because we justhad trouble getting access to the people at a charity, especially the peoplewho know things, thats changed a lot. Were still not exactly where wewant to be, but we had good luck getting in touch with, you know, gettingon the phone with this program person who knows the answers to thequestions were asking because hes [inaudible 0:08:16] or hes the CEOsometimes, which is exactly, no doubt, the person to talk to.


5/40

[0:08:22] So that first conversation is going to be very open ended andwere basically trying to approach the charity on its own terms. The basicquestions were asking, What do you do? How do you know whether itsgoing well or poorly? How cost effective is it? And questions related to

that and then, Whats the room for more funding? i.e., What do you dowith that next dollar? Because for a lot of charities the stuff that theyvedone and the stuff that theyll that theyll advertise in not the same as thestuff that theyre going to do with the next dollar that they get. Thats beenan increasing issue for us and I think its a pretty important one.

[0:08:56] So well ask those questions. Well also ask whatever othercritical questions that we can, we definitely have a lot of them [inaudible0:09:03] and in the course of that conversation well usually end up with thedocuments because someone will say, Well the best information that wehave. And well say, Do you have that written down, could you send it?

So well do that, we ask for the document, get them, review them, createinternal review where its saying what the charitys answers are to all of ourcritical questions, and what we view as the [inaudible 0:9:24]. Thats whatweve been doing for most of the year.

[0:09:28] So the next step after that is to generate rankings, whos our topcharity? And we dont know, frankly, exactly how were going to it yetbecause we are dealing with a lot of different kinds of organizations. Youcant do the same kind of apples to apples comparisons without using a[inaudible 0:09:42]. Were not sure exactly how its going to go. One thingthat we have done this year and experimented with and it might end upplaying a role in our process is we just had everyone of the staff andsometimes its just me and Elie trying to write down the 30 charities youwould give to today if you had to in order of which one you would be mostexcited to give to, then well have a discussion. So at the very least thats agood discussion connecting to them, you can just sort of talk about thingsand exploring our disagreements and that sort of exercise may end upplaying a role in how we do the final end of year ranking thats bringing[inaudible 0:10:19] a [refresher] around Thanksgiving. Right now we feelgood enough about the top rated charities that are up there VillageReachand I think its okay that the money thats going toward our website now isgoing to them, but we know, predictively from our past years, that theresgoing to be way more money coming through our site in December. Andbecause we dont believe that VillageReach has the room for more fundingto deal with that, thats when we really want to make sure we do a refreshand get our top charities out.

[0:10:44] So, thats basically where we stand. I going to pause now as wedo have some other topics, but this is the time basically that Id like to hear;A, any random burning questions that people have and B, just talk about our


6/40

process and talk about what else we might be doing besides that charity.[inaudible 0:11:07].


7/40

AUDIENCE QUESTIONS ON RESEARCH PROCESS [0:11:10-0:40:28]

[inaudible audience member] there have to be thousands, hundreds of thousands

Holden: Yeah.

Audience Member: I didnt understand how you started the ranking [inaudible 0:11:13]

Holden: [0:11:13] Yeah. So first off, were looking at developing world aid, which issometimes its very [inaudible 0:11:19] an area that we think is mostpromising is the one that were focused on for now. Were usually lookingfor charities of a certain size and thats how we filter tax records, whenwere going through tax records. We will look at a smaller organization ifits recommended to us. So in effect we done a tax record search to find asmany as we can and then its just like we have a referral form on our

website, anyone who [inaudible 0:11:44] to charity so its in the range[inaudible 0:11:48].

Elie: [0:11:51] So we just try to use a reasonable process to seed the list withorganizations that are more likely to succeed in our process than not andthat means ones that reach out to us and say, We understand what you aredoing GiveWell, we think were a good fit. You should take a look at us.Well, we look at them. And ones that are funded by other funders thatweve talked to and respect, well we take a look at all of those and then weend up with this long list and we use the flags Holden talked about toprioritize among those, which organizations well talk to on the phone or gomeet, and which organizations we have to just hold for another time.

Audience Member: [0:12:23] Im sorry, so when you talked about tax payers, youre talkingabout US tax payers. So youre really only talking about charities registeredin the United States [inaudible 0:12:31].

Elie: [12:32] No, I mean were open, the Small Enterprise Foundation is arecommended organization on our website. Theyre not a registered charityin the US and donors can give right to them or give to GiveWell and wepass it on if they want the tax deduction so its not a strict criterion that theymust meet for us to consider them, but we start there since the vast majorityof the people who use our research are US folks.

Audience Member: [inaudible]

Elie: Yeah.

Audience Member: [inaudible 0:13:19]


8/40

Audience Member: [0:13:14] Id like to know why were switching from this three criteria[inaudible 0:12:28] report like what Robert said. I mean, Im not really,those criteria you had in mind, like for example [inaudible 0:13:34].

Holden: [0:13:37] Oh, proven, cost effective, and scalable. I mean pretty much were

still more . . .


Holden: [0:13:43] Yes, its more open ended and the reason is that: A we reallywant to find something we feel great about so anything that [inaudible0:13:51]. So any area where we can look or anything we can do that wethink might reasonably correct an issue. We also just been doing this for awhile so we have more [inaudible 0:14:00]. So initially we when we startedoff we were just like, Whats the easiest possible thing you could use?And thats what we came up with. Now weve been doing this stuff for a

long time, but then again its a relatively short time so weve got the criticalquestions that we feel the need to approach different types of organizations.

Elie: [0:14:19] It doesnt make sense to us anymore to say, Just becausesomeones doing medical research we wont look at it because its notproven, or something about it is not proven. Its just that was a reasonablefirst step, but it doesnt make sense for GiveWell now to say, We cantlook at that thing, it doesnt meet our criteria.


Audience Member: [0:14:50] I wondering about, so I thought on the handouts you have the listof the different things that you were flagging and then you said, Amongthe things that we flag we rank them in terms of subjective probability ofwhat we think is most likely to meet our , or appear to be our toprecommended charity. Im wondering about how youre thinking aboutthings that have potential for getting, either a lot more funding, that have alot of room for more funding, or potential for a wide appeal, how that ranksinto things? So I mean, this is sort of like a cheap shot type question, but ifsomething might be ten times better, or something like that, you want to bespending more time investigating that than trying to figure out which oneslike 10 percent better than the other ones. It seems like if I was going to listall of the things to investigate I would want to rank them in terms of theexpected payoff of investigating them. So whatever had the highestexpected payoff Id be wanting to look into that and among the factors thatwould be relevant to what would have the highest expected payoff if werecommended it as a top charity.

Holden: [inaudible 0:16:08]


9/40

Audience Member: [0:16:10] Yeah, exactly. So Im wondering how you think about that kind ofthing and whether thats entering into the process. So you might want morethan just room for another $5 million or more funding. If something couldhave $50 million worth of more funding you could use it for longer and youmight draw in bigger donors, you might end up doing more good with that

type of thing.

Holden: [0:16:36] I think thats something that well eventually, or that hopefullyeventually, will be part of what were doing. I dont think its really relevantright now, there are a couple of reasons. There are two reasons that youmight say a charity might going to get more money from doing that so werecommend it. One is that it provides more room for more funding and theother is that it just [more of a feel good sort of thing].

The second criteria I just dont think is very relevant to us and our audience.I may be wrong about this, But I just dont see, and I dont want [to be rude

because were buying very much] but the nature of the charity that yourecommend I see as relying more on the strength of [inaudible 0:17:13]recommended because of growing on a business because of what they careabout, just accomplishing good is not [inaudible 0:17:19].

For room for more money, I think thats something that well [inaudible0:17:22] but thats more an issue when we think that the average charity welook at is not going to have enough room for funding [inaudible 0:17:34].Thatll be a day when we know more about more funding than we do nowand then [inaudible 0:17:39]

Elie: Yeah, the plays it now, the process, is just one the scale of the money weexpect to move to charities so if theres an organization that could onlyrealistically take in $50,000 in the next year that is not one that it makessense for us to put a lot of our time into because its not going to have a bigpayoff in terms of how much it can take in.

Audience Member: [0:18:03] Yeah. I was just thinking that the more extreme version of thatwhich is maybe the other thing there, just the more general point is youreranking things in terms of whats most likely to pay off, you might bepaying more attention to what could be a little bit better than the alternative,maybe 50% better or 25% better than the alternative, unless its somethingthat might be 5 times better than that. So ranking things in terms of whatsmost likely to be your top charity is not the same as ranking things in termsof what has the greatest expected impact. Its hard to do the second thing,but if theres some ways you could use intuition better track that that seemslike a game.

Holden: [0:18:46] I think that conceptions right, I think that our, like, when I sayoutstanding charity I think thats a high art of theatre, so Im just thinking of


10/40

how I would define that more [inaudible 0:18:59] so I dont really havemuch more to say to that. Were looking for a charity that we just feel greatabout so Im not really distinguishing, Well this one I might feel, like,really, really, really great in. Its the [probability that you feel really greatabout its a pretty good measure I think [inaudible 0:19:16].

Audience Member: How are you doing so far with kind of preparing [inaudible 0:19:43]

Holden: [0:19:47] Yeah, this is a tough thing to do. One approach to that is try andconvert everything into the same terms, we figured that is not a good idea.Weve been looking at a lot of work that people do in that kind of categoryof trying to reduce every charity into the same terms, we dont feel that itsthat hot a value so what we try and do instead is just take every charity onits own terms, ask the critical questions, ask, you know, we state thequestions, What do you do with the other [inaudible 0:20:15], how costeffective is it? Is there room for more funding? Answer that and create the

internal review, read them, and then the real step about deciding who wewould prefer is to check them, looking at it and saying, This is why Iwould feel good about giving, and then were going to have a conversationso that it does get, I mean, theres a lot of fat that can go off, but that theresa lot of analysis to do so these conversations are not just kind of like peopleshouting [inaudible 0:20:38] at each other and not getting anywhere, buttheyre starting from this subjective information that we take, I like thisone better. I like that one better. but theres only four of us

Elie: [0:20:49] If you look at our charity reviews, theyre not saying, Heres allthis objective data and therefore it leads you to this objective conclusionthats totally obvious. Its more putting together the facts with oursubjective judgment about what those facts mean and that leads to ajudgment. Then its the role of us as the staff, the donors, and the peoplereading the material online to say, Well, I have a different interpretation ofwhat those facts lead to.

Audience member: [inaudible 0:21:24]

Elie: [0:21:36] Well I mean were very different, the organization here thatKarens talking about it Innovations for Poverty Action. Theyre out ofYale University and they do randomized control trials of developmentprograms in the developing world. So theyre the ones that lead to therandomized control trials of microfinance programs so if we give loans topoor people what impact does that have on their standard of living. Buttheyre managing the trial, theyre checking whether that works and theyrebuilding that body of knowledge that we end up relying on when we go outand look at a particular microfinance institute and say, Well, are they doingthings that are likely to be working.


11/40


Elie: [0:22:25] Right, it is.

Audience Member: [0:22:32] You partially addressed this a moment ago, but it sounds like

youre talking that the typical GiveWell donor is to click cause [blind] andits somebody looking to do the most good and [inaudible 0:22:41] Sarah,do I have that right?

Elie: [0:22:43]Thats what we believe today.

Audience Member: Thats what you believe today, okay.

Holden: Normally we recommend something when we feel great about it becausewere like [inaudible 0:22:49].

Audience Member: I guess I was asking is that do you have a lot of people asking to give to aparticular sector, for example breast cancer research, that has not beenaddressed by you yet?

Elie: [0:23:01] Weve always had people ask us about that type of thing, I wantto know the best New York City organization. I want the best breast cancer.It's the best malarial organization.

Audience Member: [0:23:09] Is there a common one that comes to mind that has been helped?

Elie: [0:23:13] I think maybe a year ago or a year and a half ago if you asked uswhat the long term vision for GiveWell was it was creating that menu for allthese donors that have their cause purposes but not their charity purposesand the more weve been around the less weve seen that there are actuallypeople who want to do that, that will give based on those causes thattheyve laid out. One of the main pieces of evidence for that was the moneythat went to our recommended charities in 2010 where we have a menu, wehad VillageReach doing immunizations and the Against Malaria Foundationdoing malaria, and we had the microfinance one, and we had the US ones,but the vast majority of the money that went through the website went toour number one ranked charity, it went to VillageReach. Thats not ademonstration of the fact that there are not these cause specific donors, butmy intuition is that theres a lot of people who hear what GiveWell doesand what its about and the way that they relate to it is, Well, I care aboutthis cause so if only you only covered my cause, whether its malaria orwhether its homelessness in Saint Louis, well then I would use yourresearch to decide where to give. But I think often that cause is actuallydriven by an organization that they may already be connected to and theyrenot really looking for the research thats going to allow them to sortbetween our organizations.


12/40

Holden: [0:24:23] Weve got a whole blog post on this topic [URL:http://blog.givewell.org/2011/02/04/givewells-annual-self-evaluation-and-plan-a-big-picture-change-in-priorities/ ], but one wonders from our, if youlook at our most recent annual report, self evaluation between two of our

[inaudible 0:24:35] a big picture changing priorities and thats where we layout the numbers of why we [believe] in those things. Then we did anotherpost called Profile of the GiveWell Customers [URL:http://blog.givewell.org/2011/06/02/profile-of-a-givewell-customer/ ]wherewe kind of talk about the kind of person that [inaudible 0:24:47] ourresearch.

Audience Member: [0:24:54] Whats the rough number of percentage to VillageReach, veryroughly.

Elie: [0:24:58] There was like two-thirds in 2010 and the next largest one wasAgainst Malaria Foundation, which had the benefit of being a registeredcharity in the UK and Australia, and Canada. So I think that was a bigdriver of donors that want to give to one of your highly rated organizations,I can get the tax deduction so Ill give.

Audience Member: [0:25:20] Do you take into account the [inaudible 0:25:28]. Geoengineering,or something that would be really politically controversial, would yourecommend it?

Elie: Political party for example.

Audience Member: Yeah, something like that.

Holden: [0:26:04] First off, complicating [inaudible 0:26:06] were not going torecommend involvement in a conflict of interest, it undermines ourcredibility so that absolutely is a rule. As far as being controversial, this issomething that by focusing on what we focused on I think weve largelyopened the gate, just in terms of the cause. I also think that theres alegitimate argument that youve got to have a higher bar here for somethingcontroversial and [inaudible 0:26:32] its not a public relations charity. Sothats a couple of things we do [inaudible 0:26:39]. I dont think we wouldwant to avoid recommending a charity because they might be controversialand thats kind of what were here for, you know, [either one or two] are notplaying their own game or just, I dont care, I just want to do the mostgood. [inaudible 0:26:54] We may someday run into that kind of conflictand we might even handle it by privately {bringing} money for charity andpublicly doing another, but I dont really expect to run into that anytimesoon, because of [inaudible 0:27:14].
http://blog.givewell.org/2011/02/04/givewells-annual-self-evaluation-and-plan-a-big-picture-change-in-priorities/http://blog.givewell.org/2011/02/04/givewells-annual-self-evaluation-and-plan-a-big-picture-change-in-priorities/http://blog.givewell.org/2011/06/02/profile-of-a-givewell-customer/http://blog.givewell.org/2011/02/04/givewells-annual-self-evaluation-and-plan-a-big-picture-change-in-priorities/http://blog.givewell.org/2011/02/04/givewells-annual-self-evaluation-and-plan-a-big-picture-change-in-priorities/http://blog.givewell.org/2011/06/02/profile-of-a-givewell-customer/


13/40

Audience Member: [inaudible 27:18]

Holden: I just think GiveWell was [inaudible 0:27:35] were we do [inaudible0:27:42] so its not, its like GiveWell [??] and what differentiatesGiveWell is our kind of refusal to play those games so I dont really think

that we would run into those tradeoffs, but Im not ruling it out. I can tellyou right now its happening now. Theres no charity that Im not tellingyou about that I feel really great on [inaudible 0:28:08].

Audience Member: [inaudible 28:14]

Elie: [0:28:52] So theres two parts of that; how do we deal with them and thenwhat have we seen from the organization. Weve definitely found it a loteasier to get in touch with organizations, to talk to the people that are goingto give us the most meaningful information and I think largely becausepeople are starting to know the name GiveWell, certainly more than they

did four years ago, theres a carrot for organizations that participate becausetheres funding. I think last year was the first year there was reallysignificant funding to an organization that got other organizationsattentions so we have charities reaching out to us, when we reach out tothem they often want to talk.

[0:29:30] We still have organizations, and some of them are the ones thatdeclined, that dont want to participate. That could be for a variety ofreasons, whether its because they dont think theyll do well in our processor they dont think its worth their time. The piece of the process that wehave worked very hard on over the last few years and has evolved a greatdeal is how we deal with the organizations that dont want to participate inour process. A key factor for us is that we dont want there ever to be anincentive for a charity to not participate because of what we might shareabout them after they go through the review process. So we dont someoneto have the incentive to just say, ignore our email, ignore our phone callbecause they think will be better for them at the end of the day. So anyorganizations that we contact we tell them that we will put their name onour website, if they dont want to participate well write that they declinedto participate, but we wont go into the details of what they shared with usand theyll always have the option if they go through the whole process ofat the end of the entire thing that we go through just having us put up that,Declined to fully participate in our process. So thats the way weve dealtwith it right now; to offer the organizations the option to pull out at the verylast moment so they dont have to worry about causing harm by engagingwith us.

[0:30:45] Does it say something bad about the organization? I dont reallyknow, theres certainly some big name organizations that havent been asforthcoming with us as I would have thought. They seem to want to engage


14/40

from a time perspective and then only once they saw our questions theysaid, Oh, we dont want to participate. I think thats certainly concerning,but for a lot of the ones that are declining to participate I think its just aslikely that theyre small and dont have the time and dont think that ourrecommendation is going to be worth it for them.

Holden: [0:31:15] Theres a number of [inaudible 0:31:17] and I think thats theimportant thing for GiveWell and thats why we try and track our moneymoved because the more we can demonstrate that our recommendation isworth something the more incentive [inaudible 0:31:24] for charities to giveto us.

Audience Member: [0:31:29] When you start process it sounds fairly balanced in terms of tryingto find a really great charity in a [inaudible 0:31:40] In terms of otherorganizations out there, the foundations like the Gates Foundation, that do alot of work in the same realm and probably have a lot of information or

insight, how much have you been able to leverage off of what theyvealready done to get a step or two ahead in your diligence process?

Holden: [0:31:59] Some, not as much as I would like. Hey, I founded this foundationsort of offhand they dont share information. Even when they do off therecord there are often time the ability to share stuff so its been an issue. Ithink were, again [inaudible 0:32:18] and its getting easier for us to get intouch with people. So we have talked to various foundations including, Iwould say, the ones that are the most on topic maybe not all of them and wehave [inaudible 0:32:34]. But a lot of times people [inaudible 0:32:39] andalso theyre playing a different game, doing their different things and[inaudible 0:32:43] A, because they tend to fund their own projects , whichis something that weve been getting to do as a [inaudible 0:32:50 to 32:59]and then they also, a lot of what they do is kind of like kind of narrowfocused area, this is just kind of a generalization and I dont think [inaudible0:33:07] I definitely think that there are foundations that theyll take a focus[inaudible 0:33:12] including their own and then within that its kind of likeyoure not doing [inaudible 0:33:18] narrowing down what GiveWellstrying to do. Its a very different approach. So the short story is wedefinitely talk to foundations and definitely learn things from them andwere ramping that up more over time as we [inaudible 0:33:29].

Elie: [0:33:31] But to me that fact that foundations have all this informationabout all these organizations that we really care about, that we need to go tothose organizations ourselves and talk to them, ask them the questions, askfor documentation. I wish that foundations would just share the informationit would make everyones life a lot easier. So the level that were at now isthat we can talk about the specific organizations or approaches, but not justhave all that information out there, which would certainly speed things up agreat deal.


15/40

Holden: [0:33:59] I mean, if someone wants to understand our reasoning, you know,[and thinking the things through 34:03] as much as they want we wouldntsay it would be the same as, there is this dynamic that I find very frustratingwhere you go to a foundation and you say, Will you tell us about this

charity? And they say, No, we share a grant with [inaudible 0:34:15] theycan go to the charity and you ask those questions and theyre like, Well thebest way to answer your question is to send a proposal, you send it to thefoundation. So its a presumptuous problem, the foundations, its theirdefault that theyre not going to share anything and they claim [inaudible0:34:31] to charities [inaudible 0:34:33].

Audience Member: [inaudible 34:35 to 34:41] charities that you feel great about, could you saya little bit more about how much of that [inaudible 0:34:46] charities thatyou really feel great about versus [inaudible 0:34:50]

Holden: [0:34:51] Yeah, and again, I was partially referring to some blog posts that Imention earlier. One of them is in our annual reports of the major change inpriorities [URL: http://blog.givewell.org/2011/02/04/givewells-annual-self-evaluation-and-plan-a-big-picture-change-in-priorities/ ] and the other oneis called Profiles of GiveWell Customers, [URL:http://blog.givewell.org/2011/06/02/profile-of-a-givewell-customer/ ] thatsa lot of our reasoning. In brief, its a combination of most of the moneyflowing through the [inaudible 0:35:11] flowing to the number one charitythat seems to be the pattern [inaudible 0:35:15] and also just for oursatisfaction [with this] project, we dont like raising money for charities thatwe think are like, Well theyre okay, but we have a lot of concerns. Wejust feel much more comfortable being aggressive in fund raising forcharities who are like these ones are doing a lot of good. Also, we calibrateour fundraising that way so last year was when we kind of said thatVillageReach was the only one we feel really good about so lets reallypush for that one call [inaudible 0:35:47] this is the time to give. This yearIm not sure well be doing that, you know, we may just put [inaudible0:35:52] and say, If you want to give to this charity this year, but were notgoing to be like begging people to [inaudible 0:35:57]

Elie: [0:35:57] Also, practically speaking the way that our process changes iswere still looking at lots of organizations its just the goal is to find the onethat we feel best about and the thing thats changed is saying, We are nowgoing to do a project on US education, or a project on microfinance just forthe sake of having a charity in that silo.

Audience Member: [inaudible 0:36:18] do you think it would be accurate to say that rather thanthe one you feel best about theres a very small percentage you feel goodabout and that you had [inaudible 0:36:27]
http://blog.givewell.org/2011/02/04/givewells-annual-self-evaluation-and-plan-a-big-picture-change-in-priorities/http://blog.givewell.org/2011/02/04/givewells-annual-self-evaluation-and-plan-a-big-picture-change-in-priorities/http://blog.givewell.org/2011/06/02/profile-of-a-givewell-customer/http://blog.givewell.org/2011/02/04/givewells-annual-self-evaluation-and-plan-a-big-picture-change-in-priorities/http://blog.givewell.org/2011/02/04/givewells-annual-self-evaluation-and-plan-a-big-picture-change-in-priorities/http://blog.givewell.org/2011/06/02/profile-of-a-givewell-customer/


16/40

Holden: [inaudible 0:36:30] Yeah, I think we do a good job of staying up on theresearch. That was really where we did the big push up at was in our kind of2009 process, we kind of read everything we could. Since then its more onan as needed so if something new comes out thats relevant well read it.Also sometimes well end up with a new research and its like, Now

[inaudible 0:37:16] now we need understand more about [inaudible 0:37:18]so then well go read it. But I think that that, I mean, yeah, I think we couldstop that part of our process because thats not something that like certainlysomething that Elie and I do not spend a lot of time on anymore, but I thinkwere doing a good being current on the research, knowing what it says andwhat it doesnt say, and having a list of the programs that we are the mostexcited about based on the research. [inaudible 0:37:42].

Audience Member: [0:37:43] So youve had one out for four years [inaudible 0:37:47] and eventhough [inaudible 0:37:49 to 37:52] Can you tell me a little about why[inaudible 0:37:54] what does that tell you about, either giving to charities

or how hard it is to give, or what would you make of that?

Holden: [0:38:04] Right. So I wrote a blog post called Why We Should ExpectGood Giving to Be Hard and that is part of what Im thinking about here,where if you buy, its actually a lot to ask when you step back and thinkabout it, that youre going to give the thousand dollars, $10 thousand checkand then it [inaudible 0:38:19] because you kind of got a lot of people inline ahead of you because youve got to find the best thing you can do. So Ithink thats part of the issue and I think it just, you know, youve gottradeoffs because in the US its easier to see what youre doing, but its awealthy country and its probably going to be very complicated and veryhard to solve and very [inaudible 0:38:44] about how to solve them andwere kind of attracted to the developing world because there does seem tobe a lot of stuff, it just works if we [inaudible 0:38:52] anymore of it. Buteven there now were being to see a whole new set of reasons to beconcerned and reasons to be skeptical [inaudible 0:38:59] the environment[that] we dont really understand and ] the CEOs of the charity often dontunderstand very well so now you have to find ways to be confident in whatyoure doing in an environment that you dont understand very well.

So its just inherently difficult and when you first think about giving moneyto help people [inaudible 0:39:20] but with the level weve [inaudible0:39:23] for and with the level of confidence were looking for its veryhard. So thats part of it and the other part of it, I think, is that the charitableis the function. Its not set up to serve individual donors so even for thethings that [inaudible 0:39:37] I think distributing bednets [] or making surethat everyone can get free treatment for [inaudible 0:39:45]. Its just likethis vehicle out there, that charities arent there that will just take youmoney and put into that. I dont know exactly why, I think a lot of time thatstuff has to happen and you kind of chuckle. Its like very large projects that


17/40

are tens of millions of dollars that have collaboration from the governmentand once you do it you basically cover as many people as you can. So Ithink when youre standing back from the charitable world it kind of lookslike the situation where every $10 buys another bed net or when you getinto the reality of it theres both real challenges and theres like [inaudible

0:40:23] where the ecosystem isnt set up to serve the kind of person that aGiveWell customer is [thats what weve been able to change.]


18/40

POSSIBLE CHARITIES FOR 2011 RECOMMENDATION [0:40:30-0:54:29]

Elie: [0:40:29] So lets just pause. If you have a question that you havent askedus note it down and well take more questions in a few minutes. I justwanted to talk about some of the specific organizations weve been looking

at this year, both the ones that we think are most likely to be number one orbe in the top few percent that we think are really great, but also a few thatare organizations that our process with them lead to particularly interestinginsights about what theyre doing and the likelihood that its working.

[0:41:01] So one organization, its the Against Malaria Foundation orAMF, they are an organization that has been highly rated by us for a fewyears, I mentioned them earlier. Their focus is distributing bed nets in thedeveloping world. The role that they play is to ship the nets, identifyorganizations that are going to distribute nets to communities that needthem, and then buy and ship the nets to those organizations. So AMF itself

is not on the ground actually handing out the nets. Theyre a relatively smalloperation. Theyre basically run by a single guy out of his home in London,but their program is very simple, they accept applications from charities,they review the applications in conjunction with a board of malariaacademics who are evaluating whether the proposal is reasonable and theysend that to the organizations that implementing them.

[0:41:50] So one of the things that we particularly like about thisorganization, is what Holden just said, that bed nets to prevent mosquitobites that cause malaria are a particularly effective way of saving lives inthe developing world. Their track record is quite strong, both from multiple,small scale randomized control trials, but also multiple country leveldistribution campaigns that reduce malaria mortality. And so that its a veryinexpensive intervention and so that all says, You guys are doingsomething very simple. Theyre just trying to get the things that peopleneed into their hands so that they can save their lives. Theyre also amongthe more transparent organizations that weve seen, particularly in this area.They post pictures from every net distribution so you can go on theirwebsite, see distribution, and see all these photos of people getting the netsand hanging the nets up in their house. Thats a level of transparency wehavent seen from other organizations doing similar things are not even thattype of thing.

[0:42:49] The biggest question we have about them right now is the degreeto which their future activities are going to be in line with their pastactivities. So as theyve grown and as theyve raised more money theyveshifted from excepting proposals from organizations on the ground tothemselves reaching out to the Ministry of Health in Malawi or Mali, anAfrican country, and saying, We have a million dollars we want you toparticipate in your bed net campaign. Thats something that is new for


19/40

them and we dont have as good a sense yet whether theyll be able toimplement a program like that effectively because, while its still nets its adifferent type of thing.

[0:43:28] So thats the Against Malaria Foundation.

Another organization, and all the ones Im going to mention are in thesesheets, my recommendations on the first sheet at the top. The second one isan organization called PATH, theyre near the bottom of the first page.Theyre a medical research and technology organization so they are theones that are the implementing partner for a lot of the Gates Foundationvaccine projects, so theyre the ones that did the research for Rotaviruswhich is a diarrhea vaccine, malaria vaccine, Meningitis-A vaccine. Theseprojects tend to be difficult for individuals to participate in because theGates Foundation comes in and they fund a $100 million project, theresreally no room for individuals. But they have a fund called the Catalyst

Fund where they take money from individual donors and they use that fundto farm out smaller projects around the company with the hope that some ofthem end up being the very successful type of projects long term that canhave a big impact.

[0:44:24] So a couple of examples of programs that have gone through theCatalyst Fund; one is the HPV vaccine, which is effective at preventingcervical cancer, long term, that started out as a PATH Catalyst Fund project.Another, they told us this, its not something that we knew, is thatVillageReach started as a PATH Catalyst Fund project. So thats whattheyre using the individual donated dollars for. Right now its very early inthe process with them, all the signs are very promising so were excitedabout learning more, but we havent seen enough of the track record and thespecifics from that Catalyst Fund process to feel good enough about themyet to know that theyre going to be highly rated.

[0:45:10] A third organizations and as we move down this list are theodds that they end up being highly rated starts to fall so a thirdorganization is Operation ASHA. Its in the middle of the first page. Theydo a tuberculosis program, largely in the slums of Delhi in India. Theyrejust an add on to the government tuberculosis program. So tuberculosis is alarge global health problem, theres increasing drug resistance totuberculosis so we have drugs that treat regular tuberculosis, but as thetuberculosis becomes drug resistant it doesnt work as well to use thosesame drugs. They Indian government is very focused on having an effectivetuberculosis control program, Operation ASHA steps in and tries to improveit. The way that they do that is they have counselors who live in the slumsand they first encourage individuals to go get tested for TB, to take theirdrugs, its a long regiment that you to undergo to be cured from TB, its asix month thing. So they make sure and track people to make sure theyre


20/40

taking their drugs and they also set up locations where people can come andtake their medicine thats closer to their home because the general way thattuberculosis drugs are administered is under observation. Its so importantthat people take their drugs because if they dont take the full course oftreatment it leads to the development of drug resistant tuberculosis. So

thats the problem that Operation ASHA is stepping in to solve.

[0:46:34] One of the unique things about them that we find particularlyinteresting is about half of the money that they receive they pay to thesecouncilors as their wages for doing the work for Operation ASHA and thecouncilors themselves are just people who live in the slums of Deli. So to usthat in and of itself is a good in a sort a worse case, so to speak. Even if theprogram is not effective youre giving money to really poor people andtheyre able to do more with the money that they have. We havent seenvery strong evidence that their program is working yet, its something thatwe continue to look into with them. One of the biggest concerns about them

that we have is that if they, say, were very successful at getting more peoplestarted on treatment, but not successful at getting more people to stay ontreatment, or they were even less successful than the regular governmentprogram at getting people to stay on treatment they could be creating morepeople who start on TB drugs, but dont complete their full regiment andthat could be very damaging because it would contribute to the developmentof drug resistant tuberculosis.

Audience Member: [0:47:42] And do you suspect thats the case or you just dont know theanswer to that?

Elie: We just dont know the answer to the question.

[0:47:49] A fourth organization, and this is a very new one and Holdenmentioned it tangentially before, its called GiveDirectly, theyre on thefirst page near the top. They do direct cash transfers to individuals in thedeveloping world and that is in some ways a very intuitively appealingintervention because you say, Heres poor people. Well what can we do forpoor people? We can just give them money and then let them make thedecision about how best to use the funds that we give them. You couldgive someone a cow, or you could give someone medical care, or you couldjust give them the money, and if thats what they want theyll choose to buyit. So we find this very interesting. Something thats unique about them,well theres two things; one, that other organizations that weve seen thathave done cash transfers are transferring a very low percentage of the fundsthey receive to the people. So theyre doing a lot of training and mentoringand they end up only putting about 25% of the money they receive into thehands of the people who need it. In this case these guys are trying totransfer, they say, 90% of the money into the hands of the people that needit. The other pretty interesting thing about them is theyre economists and


21/40

they have set up a randomized control trial for the very implementation ofthis program and theyve also preregistered the trial. So theyve said,These are the measures were going to look at, this is what were going tomeasure, these are the effects we intend to see or expect to see andtherefore we think its less likely that theyll be able to data mine after fact

and show results that werent really there.

[0:49:13] The thing with them is theyre extremely new, they launched theirwebsite a couple of months ago. Something in particular that concerns us iswe asked them about their room for more funding and they said, Well,were only a few hundred thousand dollars now, but we could take in a$100 million in the next year if we got that. That doesnt seem quite right.Theres going to be challenges of scaling up a program so its concerning tous. We put weight when we talk to the heads of charities about howconcerned they seem to be about the potential problems that theyll face inthe programs that theyre running.

Audience Member: [inaudible 0:49:56] problems of how people are using that money?

Elie: [0:49:57] I think so.


Elie: [0:50:07] Yeah, theyve published what theyre going to track and howtheyre going to track it. Its difficult to know how people spend money. Ifsomeone came in and surveyed me a week later Id think it would be hard tosay what are all the things Ive spent money on. So one of the thingsbecause of this organization weve done some research this summer, andStephanies done most of it, on the research base for cash transfer programs,what do they day? What do we know? What works? What are the differenttypes of programs and what can you expect to get from them.

Holden: [0:50:35] Ive got a side point on that. Theyre surveying [inaudible0:50:39] that they share with the survey theyre going to do, theyre reallyexpensive, they just added so many questions and thatll be interesting forus because were not academic. I wonder how many of these studies ,usually they kind of show you line five metrics, whether those are comingfrom a survey is the important question. This is the type of thing that wekeep in mind and thats why were [inaudible 0:51:00] because we said,Youre doing a study. It would be good if, like, before the datas in youshared your analysis, your survey, your hypothesis, so that theres lesssuspicion at the end. They did that, which was [inaudible 0:51:12].

Audience Member: Do they give the money in response to disasters or in sort of the commonpoverty [inaudible 0:51:18]


22/40

Elie: [0:51:20] Right now its just common poverty situation. I think thats their intention, its not about going into disaster areas. Were excited about thembecause it seems like the type of organization that should exist. Thereshould be someone whos just trying to do the most basic, almost thebenchmark charity program. You know, theres poor people, lets give them

money and let them decide how to spend it.

[0:51:42] All right. So two more organizations that I just want to touch on.So those first four I put them all in the category of promising, were reallyinterested in them. They could potentially be the top of our list at the end ofthe year and maybe in the order that I went. It would be the likelihood thatthey end up there.

Holden: [0:51:57] Theyre not our top four, so theyre just four examples ofpotential [inaudible 0:52:02].

Elie: [0:52:02] Yeah, that is true. So theres two others I just want to touch on,which are on the other two pages. Theyre ones who I think our process leadto some interesting insights.

[0:52:13] The final organization I want to talk about is Freedom fromHunger. So Freedom from Hunger works in the cause of microfinance. Sobroadly microfinance is providing loans and other financial services to verypoor people in the developing world. Theres microfinance banks, theyretaking in capital, they actually give out loans. Theres also an organizationlike Freedom which provide add-ons to the loan. So they create educationsprograms and other types of add-ons to train people and teach them so thatthe people who are taking out the loans can use them more effectively.

[0:52:48] Something that we particularly interesting about Freedom fromHunger is theyve worked with the researchers at Yale, and Innovations forPoverty Action to do these randomized control trials of their programs sothey certain seem to have a commitment to assessing whether the programsthat theyre implementing are working.

[0:53:04] So went to them, we talked to the CEO and two things reallystruck us; first and this is just based on looking at the trials themselves the evidence itself is not all that compelling that the programs are making abig difference, that theyre really working. You could have trials that seemreally effective and ones that seem less so and these were certainly in thecategory of ones that seemed less effective. The second piece is that theirfocus is developing the curricula itself, training the staff of the microfinancebanks to implement the program, but theyre really not there monitoringwhether the programs being implemented as they intended or seeing thelong term effects, or even the short term effects of the program. They seetheir focus very much on scaling up and getting more people access to their


23/40

programs as opposed to monitoring the success of their programs anddetermining whether theyre working. So we had some debate with the CEOon our blog and posted on another blog as well [URL: not sure I can findblog posts, but there is stuff on the GiveWell webpage:http://givewell.org/international/charities/freedom-from-hunger ]. To me it

comes down to a difference of philosophy, from his perspective scaling upis the issue because theyre effective and from our perspective we feel likethe jurys still out and it makes more sense to be monitoring the quality andassessing whether the programs are effective before going through that scaleup process.

[0:54:21] So those are six organizations were looking at. Some of thatinformation is already on the website. The Freedom from Hunger review isone weve already published. But certainly by the time we do the refresh allof the details of all of this are going to be written up on GiveWell.

[0:54:35] So I just want to pause here again for questions, whether itsabout any of these organizations, any of the other organizations on the list,any charities that arent on the list, but you think really should be and weshould be checking them out, now is a good time.
http://givewell.org/international/charities/freedom-from-hungerhttp://givewell.org/international/charities/freedom-from-hunger


24/40

AUDIENCE QUESTIONS ON 2011 POSSIBLE CHARITIES [0:54:50-END]

Audience Member: [0:54:50] I was going to ask why you chose to highlight those six just now,to just give us a flavor of the types of organizations or was there a reasoning[inaudible 0:55:01].

Holden: Theyre very different from each other, they kind of provide differentcategories [inaudible 0:55:05] in the handout [inaudible 0:55:07] categories.We thought theyd be interesting to hear about, but dont take that as ourtop picks or [inaudible 0:55:12] like that. Those are six examples ofdifferent kinds of [inaudible 0:55:18].

Audience Member: [0:55:18] How do you think about part of the donation money goingtowards furthering the [inaudible 0:55:24] the learning part in particular?[inaudible 0:55:29] it sounds like an interesting idea about people who aredoing good process and seems like [inaudible 0:55:37] 90% of the money

gets passed on to the people that need it [inaudible 0:55:40].

Elie: [0:55:57] In general we feel pretty good about money spent on evaluation. Itcould be a great deal to the point where you start worrying about it if 80%of the budget is evaluation, you might start wondering why. But part of thereason for that is so few organizations seem focused on assessing whetherwhat theyre doing is working. That and so many donors seem focused onnot letting any of their money go to anything other than helping people thatthat creates a dynamic where organizations dont have the funds or even theincentive to learn whether what theyre doing is working. So certainly rightnow it seems like the right long term behavior to reward is the behavior thatis assessing whether what youre doing is working, not trying to pushpeople that . . .

Audience Member: [0:56:45] Well, Im trying to balance that against a cruising track record.[inaudible 0:56:47].

Elie: [0:57:21] Its a tough question. Theres no clear way to answer it, butcertainly the way, one thing that matters a lot about learning is whether itsgood learning or wasted learning and in that example this organization gavedirectly. Theyre running the rigorous trial, theyre preregistering the trial toprevent the extreme data mining. I think theres limited information thatexists now about how successful programs like theirs are so that seems likea pretty good thing. Then you have to get into the details of how to weighthat against the specific opportunity you have.

Audience Member: [0:57:57] And of course all the money from people that donated directly toGiveWell a good part of that is to donate money for that concern.

Elie: Right.


25/40

Holden: [0:58:05] Yeah, I agree with Fred that the factors go in opposite directions. Idont think they cancel, I think theyre [inaudible 0:58:11] that charity interms of the less established something is the more potential it has to kind ofcreate public goods of information. The more established organizations

actually work. But I think one thing to keep in mind and this is going tosound like Im contradicting, Ill leave it at that, I think almost, like the vastmajority of money that is spent on evaluation is completely wasted, in myopinion, because its not good evaluation so you spend a lot of money, youspend a lot of cash and you end up with a study and then you look at thestudy and were like [inaudible 0:58:44] should buy it and so well[inaudible 0:58:47]. And a good study, I think, like because good studies areso rare that a good study is really what we wanted to see. So pickedsomeone really credible [inaudible 0:58:57] like we really are going to learnfrom what theyre doing and it really is going to produce that information,and that informations really going to be useful, that I think thats really

valuable and cancel out a lack of track record to some degree, but most ofthe time we want to go with what works because the default assumptionwhen somebodys evaluating is that theyre wasting their money [inaudible0:59:18].

Audience Member: [0:59:20] What is your plan as far, it sound like [inaudible 0:59:22]VillageReach theyre probably going to reach a point where [inaudible0:59:24] you get your money from, theyll fall off of your recommendedlist. What is your plan for a following up and seeing how effective theyvebeen even after youve kind of [inaudible 0:59:36]. Like PSI I rememberwas one of the ones you had recommended as far as [inaudible 0:59:41].Like are you still following up with PSI and seeing how theyre [inaudible0:59:44].

Holden: [0:59:46] I guessed there would be two factors in how much we follow upwith a charity, we do it for business sources. We wouldnt follow up withsomeone just because weve surveyed them privately but we would followup with someone either if we still do [inaudible 0:59:59] or still think aboutsending a follow up with or if weve gotten a lot of money for them. Youcan [inaudible 1:00:05] the matter of understanding the types we a have andknowing if we made a good call or not [inaudible 1:00:11]. So weve got alot of [money to VillageReach for each prior standard and we are followingthem very closely. Weve been posting quarterly on [inaudible 1:00:18] andweve been really thorough with them and I think one of my favorite thingson VillageReach, like I [want to be clear], what theyre doing, and it mightwhat theyre doing may fail, and] weve known that completely since thebeginning, but one of the things that I do need to [crack] into about whattheyre doing is that I feel that [inaudible 1:00:36] and thats a very rarething in the world of charities, that you know [inaudible 1:00:40] somethingfails, you never find out and I am really convinced that if VillageReach fails


26/40

youre going to hear about it and were going to learn something and wewont have that []. And thats something were definitely doing.

Audience Member: [1:00:59] Have you guys considered, now that youre talking about movingsomething close to a million dollars potentially or more next year,

considered going to a charity and saying, We like you, we have someconcerns about your self evaluation. Would you be willing to change yourprocess and say dedicate 20% of the money that we might get moved to youdefault the valuation contingent on our rating you? Or sort of a bit more ofa close partnership where you maybe even give them advice about how theycould supply you with better statistics or do better work in self evaluating?

Holden: [1:01:40] In a sense I think were going to use something like that with thenew initiative, which Ill talk about at the end, like, will it [at least] be opento something like that? Its not that we [inaudible 1:01:51], you know; A, Ithink we generally have to really offer money to do that and be able to offer

that money and not just say, like, Well [inaudible 1:01:58] and B, its justimportant to keep who we are and what our value added is. Were notevaluators, were not researchers, were, like, basically donors trying tofigure out what to do. So what I do think we add a lot of value is looking atresearch thats coming and saying whether its credible and whether itchanges our opinion. But we dont, like, necessarily know much about howwe design research because theres always other issues, like, I know what adecent study looks, but I dont know how many things get in the way of myidea of a study when you actually get to the ground where the charitysworking. So its just important to keep that in mind. We definitely[inaudible 1:02:40] that woman how hard it is to improve it, we do that, butwe definitely have not pushed ourselves [and put them] in that way, ever.We may someday sort of like that, I think we would be more and morefocused on accountability.

Elie: [1:02:55] More of the way that we accomplish that goal is by holding outthe carrot. We say, Heres our standard, you can see whos high up. Hereswhat we think about what youve sent us. You have the opportunity in thefuture to win the money, get the grants, if you can demonstrate that youmeet that standard, so were not there to provide the funding itself rightnow, maybe we would in the future, but thats our main approach to thatproblem.

Holden: [1:03:25] Sort of in line with this conversation about high priorityorganizations that weve mentioned]. It seems to me that what seems tocoincide with that sort of priority is big emphasis on vertical programs somereally discrete programs which has more measurable outcomes. But insteadof focusing on organizations that do, what I would refer to as morecomplex, integrated stuff, like healthcare, general healthcare, primaryhealthcare, [inaudible 1:03:58]. Theyre much more interested in an


27/40

organization that would focus on a specific intervention like condoms,[inaudible 1:04:06] stuff like that.

[1:04:10]But I have a conflict of interest here, I have worked with Partnersin Health, not officially, Ive worked sort of alongside them in Rwanda.

They are a good example of an organization that originally would have gonereally well in the [dippo] evaluation so they emphasize [inaudible 1:04:25]TB and HIV care and they had a really good [inaudible 1:04:28] program. Itwas a very specific, discrete intervention with very measurable outcomes.You know [inaudible 1:04:34] that these patients [inaudible 1:04:40] have alot of other problems [inaudible 1:04:42] of other things of course andcreated a much greater priority for the integrated, you know, primary care,just children are dying from all kinds of things that are hard to do in termsfrom an intervention, a vertical intervention, like diarrhea or [inaudible1:05:03].

[1:05:02] So as it seems to me that was a conflict between what measurable,that is a tradeoff to be measurable discrete projects, sort of complex lessmeasurable outcomes to probably measurable profits. [inaudible 1:05:20 to1:05:23] the number of nurses that have been hired, the number of patientsthat are treated daily, etc., or the number of drugs that have been [inaudible1:05:29] the challenges it seems to me of a higher bar, which I think isprobably creates some problems for evaluating things, which is that youwant to know the impact, which is really difficult [inaudible 1:05:41]. Imsort of okay with it, but I think my only problem is with final conclusion Iwould draw is probably from your evaluation because probably the oppositeis what I think the money should be spent on, which are things likehealthcare and education that are quite a measure, but really important. So Idont know how to get around that. The problem is you cant do anythingwithout the data, but it seems to me is theyre just a different kind of datathat you would need, some sort of transparent sense of, well their objectiveis probably different] the data will mean were going to be, its differentgiven , I dont know.

Yeah, and thats a long monologue.


Holden: [1:06:23] I appreciate it and weve had this discussion a little before, but Ithink we just wanted to have it with everyone here. Theres actually twoissues , so one is measures are we biased towards whats measureable andwhats noble and two is how do we approach something like Partners inHealth, which I think those are very different things because I think Partnersin Health actually does pretty well by our criteria, but we are biased towardswhats measurable. So just because, you know, we are, on the first question,are we biased towards whats measurable? Yes, again. And I think we dont


28/40

want to hide that, we dont want to be shy about that. I think thats a goodthing because what were trying to do is basically [inaudible 1:07:02]people who are nave. [inaudible 1:07:04] to get poor people that they knowfairly little about and I think that thats a bias towards what you knowyoure good at and what you know youre going to be able see success on,

its very comfortable there and its kind of a big part of the GiveWellphilosophy of its not about what we wish we could accomplish, its aboutwhat we can accomplish and its not taking the entire responsibility forending poverty on sort of shoulders of donors because thats not a[necessity and not possible. But rather a lot of the stuff that happens is outof our hands, its up to the individuals and were just trying to be as helpfulas we can. That means doing the things that were able to help with.

[1:07:41] So in that sense we have a bias, not necessarily towards whatsmeasurable but towards whats noble, towards whats [concrete], towards towhat we have high confidence in. And for that reason, like, you know, my

favorite sector right now is developing world health because healthtechnology is just, you know, works, I just have much higher confidence inthat stuff, in going and treating someone for HIV or, you know, givingsomeone a bed mat or towards deworming, or tuberculosis treatment. ThoseI have much more confidence in than going and showing someone how torun their business because, you know, and theyre kind of [inaudible1:08:19] and because its about helping people, its not about us, I dontwant to take the whole responsibility onto our shoulders. I want to say, Ifhealth is all we can help with, lets [inaudible 1:08:27] and let people do therest.

[1:08:32] So theres [inaudible 1:08:33]. You know, that actually leads us tobe pretty [inaudible 1:08:35] towards Partners in Health. They were arecommended organization, theyre still on our list of organizations that wewould like to know more about. Theyve been, like, not very responsivewith us. I think thats partly because of the, you know, the fact that their[inaudible 1:08:51] theres a lot going on, theyve had a lot of money come.[inaudible 1:08:56] So bottom line, I mean, we still need to haveinformation, we still need to have critical questions to get answered.[inaudible 1:09:06] anything from Partners in Health because we feelcomfortable with that, but do, I mean, we spend a lot of time on NyayaHealth which is also doing generalized healthcare. I dont think we have abias towards vertical healthcare. VillageReach is

Elie: [1:09:19] And just like Holden mentioned a good example is this review of Nyaya Health. Its a health center in rural Nepal. We got very interested inthem because of their focus on transparency and openness with their dataand what they were doing. So that allowed us to learn a lot about themreally quickly because they were so open, they would talk to us very easily.You can go to the review thats online now and see the types of questions


29/40

that were asking them and its not, Oh, it seems like youre not costeffective enough, or, We dont see the RCT that your patients of TB careare getting cured. So we were pretty excited about them, but in the end thequestions we had for them we werent able to answer.


Holden: [1:10:26] We did look into this, I think this is a good point, a good questionand basically weve asked that question. I mean, if you say to us, Whathappens next? We couldnt, conceivable perhaps, but [inaudible 1:10:38]when you save someones life its in the wrong place that they would, like,exaggerate, you know [inaudible 1:10:42] You know, weve done somework on this, theres a page on our website called Standards of Living andDevelopment and where we say everything we know about, kind of, youknow, that question, When you save a life what do you save? We feelpretty good about it. I dont disagree with you at all, I think thats

conceptually a big concern and in reality its a big concern too, but I thinkthat when you help someone deal with NPC I think that a lot of times theyare going to live 50 to 60 and maybe they wont have an occasional[inaudible 1:11:16] or wed love being able to have all the opportunities thatAmericans have, but theyre better off. And so yeah, its just an importantpoint that people, like, theres been a lot of reduction in poverty over thelast couple of hundred years and most of its our fault. So its just importantto keep that in mind, you know, we are totally in favor of, like, maximumgiving to the types of organizations, we dont want anyone to get anotherimpression. But its just, Whats your role? Its just the case that povertywill exist until we as a group stamp it out or, you know, its just a case thattheres a lot going on and its complicated and we just want to [go about] inthe way that we are attempting to help.


Elie: [1:12:35] I dont know how common it is. I think that there are a lot of organizations where theres a lot of layers of staff where they are moreeducated and more well to do before you get to the folks at the bottom whoare actually implementing the programs. We have the experience even whenwe were visiting organizations in India that theres the India staff andtheyre implementing the program, but theyre college educated people,theyre not people who would be earning a dollar a day, so to speak. Whilethat organization in India is granted it out to smaller organizations, I think, Idont know, but my guess is that theyre passing through a relatively smallamount of money. But its an interesting question and its not one thatweve really looked at very closely.

Audience Member: What do you think about GiveDirectly [inaudible 1:13:28] I read what yousaid about it about two months ago and its partly a two ways that its seems


30/40

so promising. I did see the study that they had addressed, or mentioned atleast, that this seem to say that it was being used well. I guess in part fortwo reasons, one is that . . .

Holden: [inaudible]

Audience Member: The money that had been given out directly to PATH. They didnt have astudy about that already that a percentage of it was being used, or beingsaved, it was like 2% was being saved at least, [no word about being spenton alcohol, etc. That thats partly for two reasons, one is that, I thought waspretty well shown that when people get money for no reason, like when theywin the lottery, their lives are not improved. Also, I thought that your recordwas pretty good that in microfinance why was giving away be useful, butnot loaning. I guess I wasnt really sure why that would be better.

Holden: [1:14:13] It would probably be for a couple of reasons. I mean, one, you

know, the lottery is a very different situation from people living on a dollargiven another dollar a day, its just very different ballgame. I mean, I think,theres a lot more things. Like theres a much steeper curve of, like, usingmoney to improve your life, I think, when youre more poor. Thats one ofthe intuitions I have that I think you know makes sense and then there is thiskind of, you know, this theoretical argument that why is giving peoplesomething else better than giving them money? You know. How do weknow what they need to be buying, why arent they, no, and I dont thinkthat [inaudible 1:14:49] but its an argument.

Audience Member: So whats the difference with microfinance? If giving money doesnt work,why would loaning money

Holder: I mean, I think when you loan money I think a lot of those people could endup financially worse off, its like [inaudible 1:15:33] because theyre payingvery high interest rates on that [inaudible 1:15:08] I mean definitely muchthan on US credit rates and, you know, a lot of times much higher than that.

Elie: [1:15:18] Holden visited the Small Enterprise Foundation, now a year andhalf ago, theyre a recommended micro finance organization and one of thestories that the staff told him was about a woman who was buying goodsand then selling them at cost, but buying them with a microfinance loan. Soshes taking this loan, paying the equivalent of 100% APR, which is aninsanely high rate here in America, but in selling the goods that she boughtat-cost making no money and then whole enterprise was just losing moneyand it can be hard, I think, for people to track whats happening becausethey have some many different sources of income and credit that there are, Idont think its even that implausible that there are people for whommicrofinance is causing harm.


31/40

Audience Member: [1:16:06] On the GiveDirectly, I was kind of surprised that that was makingit to this level of being a plausible contender because I was having a hardtime imaging how people would be spending, I mean, how you would get sothat these people are spending the money more efficiently than , like, thebest health and that weve managed to find. It just seemed, I would have

had a low prior that if we handed money to people that they would bespending it on things that were like better than bed nets or something likethat. And then thats just an intuition I guess. And some of the reasons Ihave for thinking that were, like, theres this argument that you talked aboutbefore, like, Well why do we know better? Are you saying we knowbetter than these people? Well, I mean, I guess I would have thought,Yeah, if you spent a bunch of time trying to figure out what the ways tospend money that helps people the most you would know better than a lot ofpeople about how to help people the most.

Holden: [1:17:16] Yeah. I mean, I guess I, I mean I disagree with that, but I do feel

better if you were really just delivering bed nets to some of those top healthinterventions. I would probably feel better about those than cash. So I dontthink its all that obvious. First off, you know, why deliver nets when youcould deliver cash, [inaudible 1:17:36] There are certainly arguments, butpresumably people do invest in their own wellbeing and their childrenswell being. Thats one side of it, but I dont know, I just [inaudible 1:17:50]it just that I believe in the pencil light of knowledge. I believe that no matterhow hard I work to understand poor people Kenya, the poor people inKenya probably still understand themselves better than I do so I think itsalmost in a very abstract way, I mean, I think that, you know, I dont evenknow about the specifics, I just feel like generally when Im looking atsomething from this distance [inaudible 1:18:13] and someone looking at itfrom up close has more power to see what will really help them. Itscertainly not impossible at all that, you know, theres somebody in Kenyawho has a great business opportunity if only they had some extra cash orthey could invest in, you know, in something that would improve theirproductivity in some way, that might include education. And once youveimproved your productivity well then youve lots you can spend that to getthe health care you need and to get the other things you need. So I dontfind it improbable in the specifics that there could be things to buy that arebetter than bed nets and also, you know, I find it just any abstract, I find itvery compelling the idea that no matter how good something looks to me,Im still very far away and the person thats up close [inaudible 1:19:01].

Audience Member: [1:19:01] Yeah, I guess Im just thinking, you know. So I said low prior, Imean, Im not saying, like, its like 1% chance of that, I dont know.Maybe, like, 10% subjective probability or something like that, that couldhappen. But, you know, were talking, like, knowledge gaps, I just dontimagine people thinking like, Ah yes, well like pay to get dewormed


32/40

now, or something like that. I just dont think people would have thatknowledge a lot of the time.

Audience members: [inaudible 1:19:33]

Holden: [1:20:33] Yes, I actually find that argument more compelling than the otherone of why health prevention, thats the troubling one I just dont feel good.[inaudible 1:20:43] I just dont feel way like I know better than the people[inaudible 1:20:47] and I think about, you know, aliens invading the Earthand how I would feel if they were running things badly [inaudible 1:20:58].

Audience members and panel members: [inaudible 1:20:59 to 1:25:50]

Elie: You make a good point, you can find other evidence so its not randomizedcontrolled trials of grants, that is, people getting double the amount ofmoney they had before and how they use it and what happens to them in

that situation.

Holden: Yeah, I mean, certainly if you just compare people who have two dollars aday versus people on one dollar a day, I mean, you would be prettyencouraged. On the other hand thats a problematic comparison becausewhen you make more money, I mean, it means that you have money, Imean, it means money, you know, you may be better nutritional [inaudible0:1:26:16].

Elie: Why dont you just talk about the last little piece in the feature and then wecan at point . . .

Holden: [inaudible 1:26:26] Okay, so yeah, so the last thing that we just want tomake sure is covered tonight is just [inaudible 1:26:31] which is that a lot ofpeople are comparing to just like giving bed nets to people and if I werechoosing between cash transfers and just giving bed nets to people bed netsto them, I know theyre getting them, and I know that [??] of malaria, Iwould take the bed nets. I dont think it would be that hard a decision and Ithink [inaudible 1:26:59] and the same with a bunch of other healthinterventions. The problem is there are interventions that I feel that wayabout and when youre doing those interventions you have a whole bunchof to identify poor people, you give them the tent and then you give themthe transfers and with the health interventions youve just made a healthintervention and Ill just name a whole bunch of other [inaudible 1:27:20].So thats why I have them listed in the as unlisted as a group that interestsus that will probably be reasonably high on our list in the scheme of things,probably [inaudible 1:27:28].



33/40

Elie: Yeah, I think theyll be on our website this year. I dont know in exactlywhat form so we wouldnt need to see their RCT just so that it could[inaudible 1:27:51]

Holden: So theyll be similar. I think they will probably be better in our rankings

after the RCP happens and thats a big [inaudible 1:27:56]

Elie: I mean, one kind of interesting exercise to go through is just to look at thelist of ones that we highlighted as the ones were looking at, ones on ourwebsite, and just, you know, to do the top 30 rankings yourselves and let usknow what we missed. I mean, not right now or if you have ideas now thatsfine, but you know, as an exercise because one of the things we find is wejust start, you know, Give Directly, well we talked about all the problemswith cash transfers, but theres a host of other organizations that we comewith all sorts of problems that seem less problematic than the ones of GiveDirectly.

Holden: So I, just to get, you know, to get the last point that we want to break outhere [inaudible 1:28:42] and then well have it up again. One of theoverriding themes here is that it has been really hard to find great charitiesthat we feel great about and were thinking about the future of GiveWelland how were going to, you know, not just with what we have because[inaudible 1:28:57] so I want to talk about a few of things that werethinking about for that.

One of the things is a new research initiative, it was kind of vaguely alludedto in the attachment as project funding. This is something that its notannounced yet, were going to announce it reasonable soon. Were going tobe calling it GiveWell Lab and the idea is just that it, you know, GiveWellwe open . So no matter what the measure, no matter what form, were opento recommending anything and that would include do a specific projectinstead of funding it through the charities. Some examples for instancemight help: A, you could fund a study; B, you know a group likeOxfam[SP] that does so many things that its impossible to trace whatunrestricted donation does. You can [inaudible 1:29:45] You can look attotally different problems, you can look into charities that dont even reallyseem to fit into and were going to allocate unlimited amounts of our time tothis as a kind of way of exploring other options and hopefully what we learnfeeds into the normal process. So thats one thing that were pretty excitedto do. I think you might find a better opportunity.

Another thing that were going to be doing and there are specific causes thatwe just, we could and should learn more about, definitely water, programstargeting orphans and vulnerable children, surgery programs and medicalresearch are all ones where we kind of [inaudible 1:30:24] weve beenlooking for research having turned up enough information, but we think that


34/40

we can learn more about these things and we think that theyre all appealingin different ways in terms of areas where there might great charities. Sothose are sectors that were thinking harder about, spending more timetalking to people, and figuring out how to evaluate them better.

Finally, I do think we want to be making with this as we grow in ourprocess and I mean both in visits to the programs on the ground and alsovisits to the offices and meeting these people. Weve just found, I dontthink Ive ever heard, I mean, I doubt if they ever recommended charitiesjust from the [inaudible] but they bring up a lot of questions that you didnthave before. You can also get a lot of questions answered more quickly ifyou can get them, you know, to be confident that with these past studies thatare often unreliable. So I think thats something that we want to be doingmore of and that may make our process richer and better responding andbetter [inaudible 1:31:25].

So those are things were thinking about doing in the future. At this pointId just, you know, weve got the night, I think we can poll people how weneed. I still think were just going to throw it open, if you want to askquestions or give comments about I just said, go ahead, or about previoustopics, or if you just want to throw out any charity, or cause, or topic, o

Documents

GiveWell August 18 2011 Research Event Transcript