Embed Size (px)
Citation preview
The Human Race -‐ and The Technology Race! George Gantz 2014
Chapter III: The Technology Race -‐ to Superintelligence?
Previous Chapters: Chapter I: The Human - Technology Interface Chapter II: What is the Human Race? Forthcoming Chapter: Chapter IV: Practical Advice - Responding to Technological Change
1. Introduction In Chapter I we provided a survey of some of the problems in the Human -‐ Technology interface. Advancing digital technology does not always, and perhaps only rarely, integrate seamlessly with the biology of its human masters. In some cases the technology may be disruptive to human aspirations and capacities. In Chapter II we examined in greater detail the questions of what it means to be human and what the goals are for the human race. Remarkably, we have made extraordinary progress, particularly in recent centuries, in terms of human population, lifespan, violence and economic well-‐being. Humanity has been thriving, in large part as a consequence of technological innovation. Will this positive trajectory continue? It seems that will depend on the trajectory of our technological advance, and particularly the trajectory of digital technologies that fuel and shape all the rest. Given the ubiquity of digital technology, will future advances lead to human thriving, or to something else.
Page 2 Digital technology is exerting a transformational influence on almost every facet of the human economic and social enterprise. Each of these facets will follow a unique trajectory. This makes “general” predictions about the future difficult if not impossible. The evolution of digital technology applications is likely to be as chaotic and seemingly unpredictable as the evolution of biological life, although on a far shorter time horizon. Yet there is one potential digital technology that is getting increasing attention, and that may, in time, drive all the rest. That is the goal of designing and building human level general intelligence. Some have called this -‐ the “Singularity.” In this chapter we will explore the basis for the optimism many are expressing for achieving artificial super-‐intelligence, as well as the concerns that are now being raised about the potential consequences of such an achievement. We will conclude with some observations about why creating a super-‐intelligence may be vastly more difficult that its promoters believe.
2. Moore’s Law In 1965, Gordon Moore, the founder of Intel Corporation, offered a theoretical projection of continuing exponential advances in computing power and capacity for the coming decades. This is now known as “Moore’s Law”, and, remarkably, Moore’s Law has, over the past 45 years, so far held true. In terms of speed, capacity and memory, our computational capabilities have continued to increase at an exponential rate. Chart 1 on the following page shows a version of Moore’s Law from Intel Corporation covering the period from 1970 to 2010. Chart 2 shows an example of Moore’s Law from Ray Kurzweil, author of the 2005 best seller The Singularity is Near – When Humans Transcend Biology, covering the period from 1900 to 2010. I’ve added a vertical line to Chart 2 in 1970 to show the corresponding period as in Chart 1. These two graphs use a similar logarithmic scale as the graphs in Chapter II – the upward sloping line reflects an exponential growth rate. Each horizontal line as you move up is a ten-‐fold increase over the line below. What these graphs show is that, using two different measures of computing power, growth has been consistently exponential over the period. This is dramatic progress. In the 1960’s and ‘70’s. Moore’s Law was considered by many to be an extremely optimistic projection, as each stage in the advancement of computing power appeared to confront increasingly more difficult constraints to further breakthroughs. Yet almost five decades later the projection continues to hold. For many it has become an article of faith that computational advances will continue to follow Moore’s Law into the foreseeable future.
Page 3 Chart 1. Moore’s Law -‐ transistor capacity 1970 to 2010 (Intel).
Chart 2. Moore’s Law -‐ calculating speed per $K -‐ 1900 to 2010 (Kurzweil)
Page 4 Chart 3 below shows a conceptual projection of technological advancement published by Time Magazine online. Along the top we see a list of major technological advances along with the time span between them: 8000 years between the agriculture revolution and the industrial revolution – another 120 years to the invention of the light bulb (the electrical revolution) – 90 more years to the moon landing – another 22 years to the world wide web and then just 9 years to the sequencing of the human genome. These big technological discoveries seem to be speeding up! The lower part of the graph shows another representation of Moore’s law, also beginning in the year 1900, with various computer inventions highlighted. The graph also continues into the future as a forecast. On the far right, the graph suggests that by 2015, computer capability in terms of speed (calculations per second) will increase to the point that it will surpass the brainpower of a mouse. By 2023, it will surpass the brainpower of a single human, and by 2045 it will surpass the brainpower of all human brains combined. As the graph proclaims: “The accelerating pace of change and exponential growth in computing power will lead to the Singularity.” Chart 3. History and Projection of Technological Advances (link)
I don’t suggest, as the Time Magazine graph apparently does, that you can equate computational speed with intelligence. Nevertheless, most experts do predict that computers will achieve human level general intelligence in this century. Some contend that the increase in computer intelligence at that point will be so fast that it will take on the appearance of vertical growth – hence the name “Singularity”.
Page 5 3. Super-‐intelligence Is it likely that our computer scientists will eventually manage to build a computer program, installed on appropriate and necessarily sophisticated hardware, that is capable of learning and has the capacity to outperform humans on every test of intelligence? In terms of learning, some computers can do that now, and the results are proving to be exciting as well as interesting. Among the interesting findings: In Switzerland, robots with learning and communication capability were trained to play a game together where they seek to find “food” and avoid “poison” -‐ these robots quickly learned that lying to their compatriots could be a successful strategy. Computer programs are now also quite capable of outperforming humans in a wide array of activities. Among the most famous examples is Deep Blue, the computer program that beat reigning World Champion Gary Kasparov at chess in 1997. The computer program Watson also beat the reigning Champion in the notoriously tricky game of Jeopardy in 2011. Some games, like GO or poker, seem to be more difficult for computers to master, but it seems to be only a matter of time until specialized computers are built that can best humans in those games, too. Computers are now also routinely being employed in what are considered expert tasks – medical diagnosis and surgery – engineering – operational and manufacturing controls – interactive telephone networks. Interactive computer programs, such as the Siri program on the iPhone, are becoming commonplace. While it may take some time for such computers to be able to pass the “Turing Test”1, it no longer seems very far-‐fetched. Intelligent, interactive computers are common in fiction -‐ in time they may seem commonplace in the daily world as well. Clearly, we are moving quickly in the direction of getting smarter computers – doesn’t it seem just a matter of time before someone builds a computer program that is smarter than its creators? What happens when we do? Some speculations:
! It is likely that a computer program with human-‐level general intelligence will be able to assimilate vast amounts of knowledge and information quickly.
! Such a program will probably have Internet links and access to many resources, potentially including robotics, satellite data and remote sensing.
! By virtue of its learning capabilities, it will be able to adjust its own programming – and perhaps its associated hardware and sensing devices -‐ in ways that will in time (perhaps quite quickly) increase its effective intelligence to levels well beyond what we can imagine.
1 Alan Turing (1902-‐1954), one of the founders of modern computing, developed a common sense notion of computer intelligence. When a human has an extensive conversation with a hidden entity and is not able to determine whether it is human or machine, the entity has passed the Turing test.
Page 6 The consensus among the experts is that once human-‐level general intelligence is achieved, that program, network or Artificial Intelligence will evolve quickly and become a super-‐intelligence. With super-‐intelligent capacity, the entity could link to and control computer networks, communication devices, hardware manufacturing facilities – essentially our entire sophisticated digital infrastructure – in order to achieve the goals that had been set for it. Potentially, this capacity will enable it to identify and overcome just about any technical or resource barriers -‐ including any barriers erected by its much less intelligent creators. How can we deal with such a machine?
4. Raising Concerns The real question may be -‐ how would such a machine deal with us, its human creators? Isaac Asimov, scientist and writer, pioneered the “laws of robotics” in 1942: The first law is that a robot may not injure or allow a human being to come to harm; The second law is that a robot must obey the orders of humans unless it violates the first law; And the third law is that a robot must protect its own existence unless that violates the First or Second Law. This is all nicely reassuring. However, these laws are a literary construct for Asimov’s science fiction stories -‐ they do not exist or apply in any governing institutions. Moreover, in the context of complex systems and advanced digital technology, these laws are going to be unfeasibly simplistic. Thankfully, to date, they have not been needed. You may also feel reassured to learn that Google, one of the largest and most innovative corporations engaged in the digital transformation, has adopted an official corporate motto -‐ “don’t be evil” – and they have instituted an AI ethics review board which we can hope will be able to enforce that motto. Whether these constraints will be effective in the context of a culture so thoroughly committed to the digital transformation is another question. In any case, the topic of super-‐intelligence is now getting significant discussion and debate. One key question is whether and how we might be able to achieve the intent of Asimov’s laws. My sense at this point is that discussion so far is limited to the expert community. It has not yet been given a lot of attention by the public, the government or the international community. Meanwhile, the notion of an evolving super-‐intelligence is firmly embedded in the computer science world – it is not just science fiction anymore. There are books, conferences, associations and investments (including some large investments from silicon valley billionaires) flowing into the field. For many, the goal is to get there as soon as possible because they believe in the progress this will deliver for humanity – or at least
Page 7 some of humanity. The opportunity for extraordinary profits may also be a motivation, albeit a less public one. There is even a new social movement called Trans-‐humanism that is promoting the concept and goal of super-‐intelligence – the idea of humanity evolving, in conjunction with digital technology, beyond the capabilities and constraints of the human body and mind. Ray Kurzweil helped popularize these concepts in his book The Singularity. Ray is 66 years old – a brilliant computer scientist and inventor now serving as Google’s Director of Engineering. He is convinced that super-‐intelligence will arrive quickly and will bring with it the effective end of human mortality. His personal goal is to make sure he lives long enough to achieve immortality – he reportedly takes hundreds of nutritional supplements daily. This is a man who is committed to his goals! Nick Bostrom is also a very smart guy and, to my mind, more circumspect than Kurzweil. He is the Director of the Future of Humanity Institute and very concerned with issues of existential risk for the human race. Earlier this year he released a book, Super-‐intelligence – Paths, Dangers, Strategies, and in recent months he has been quoted and interviewed a number of times in the media. Bostrom identifies five pathways to Super-‐intelligence, including:
∗ Artificial Intelligence – an intelligent computer program ∗ Brain Emulation – reverse engineering and simulating the brain ∗ Biological Cognition – a synthetic biological brain, also known as “wet AI” ∗ Human-‐IT Interface – symbiotic integration of smart hardware and humans ∗ Networks / Organization – an “emergent” super intelligence
He also discusses at length the key risks associated with a developing super-‐intelligence. These include concerns with Kinetics, Strategic Advantage and Cognitive Superpowers and the issues of dealing with a Super-‐intelligent Will. Kinetics refers to the possible speed and surprise that may be associated with the evolution of super-‐intelligence, once human-‐level intelligence is achieved. Bostrom suggests that super-‐intelligence could emerge in a matter of hours or days – and what emerges could be radically different from the programmers’ expectations. Even if the process took many months, Bostrom points out that the first program or entity (or the corporate or government team creating it) would have a significant strategic advantage over any subsequent efforts. Arguably, this would leave the world with single monopoly super-‐intelligence. Even if multiple efforts resulted in a variety of competing super intelligent entities, all of them would in some manner exhibit cognitive superpowers – the ability to out-‐think its human creators faster than the blink of an eye. If robots can already learn to lie, it is sobering to wonder what a super-‐intelligent computer will learn in the first few moments of its existence. These scenarios lead to the larger issue of dealing with a Super-‐intelligent will. Can we
Page 8 control what choices and actions the super intelligent machine will take. If we are able technically to impose such a control mechanism, what structure of values and ethics should be embodied in those controls, and how can we “teach” them to the entity we have created. Human values are complex and often conflicting – how do we choose what values to teach?
5. How Real is This? The scenarios outlined by Bostrom are rather frightening. Should we be concerned about this? The obvious answer is -‐ Yes. According to recent surveys of experts reported by Bostrom, there is a 50% probability of achieving human level general intelligence by 2040. Many advocates, including Kurzweil, are counting on something a lot sooner. Should we panic? I don’t think so. In the paragraphs below, I raise four factors that I believe will significantly extend the time period for the achievement of super-‐intelligence. In fact, it may prove to be impossible. These four factors are:
a) The Law of Diminishing Returns b) Human complexity c) Hard limits in math and science d) The problem of consciousness
a) The Law of Diminishing Returns The images of exponential growth that we have seen in the various charts in Chapter II and III are compelling and tend to capture the human imagination to the extent that we believe that exponential growth is a fact, and not a contingency. However, the future is always contingent, and any particular trend is contingent on limitations and constraints that may not be obvious and may not have applied in the past. In the finite world we live in, such limitations and constraints are inevitable. In economics, the reality of such constraints is reflected in the Law of Diminishing Returns, a concept attributed to Adam Smith and other early economists. The Law states that increasing a single factor of production, while holding all others constant, will at some point yield lower incremental per-‐unit returns. While we can continue over time to drive down the cost of computing, or increase computing speed or capacity per dollar, eventually the opportunity for exponential growth will be exhausted. Chart 4 below shows a simple graph (the scale is linear, not logarithmic) of exponential growth. This could be a chart showing bacterial growth in a petri dish, or the penetration
NOTE: The Singularity does not appear to require that computers acquire consciousness, but most experts assume that this will be the case.
The consequences discussed above may be the same in either case.
Page 9
Growth appears exponential
Growth slows down
of a new digital technology in the global marketplace. As long as there are no limitations in the time period considered, the growth demonstrates an exponential, accelerating trend. Chart 4. Simple exponential curve.
Chart 5 shows what begins to happen when limitations are encountered. The graph starts out looking the same as Chart 4, but then changes shape and levels out. This would happen, for example, when the bacterial colony reaches the edge of the petri dish and begins to run out of food, or when the digital technology begins to saturate the market. The growth slows down and eventually stops. The resulting graph resembles what is known as a “logistic” curve – the parameter being measured reaches a maximum point beyond which further increases are impossible. The increases have to stop – no new bacteria can fit in the petri dish – there are no more new customers for the digital technology. In case of the bacteria, the curve may actually reverse as the population collapses from lack of food. Chart 5. Example of logistic curve – growth under constraint
Page 10 The Law of Diminishing returns has other formulations that demonstrate the same principle. We often hear that you can get 80% of the way to the goal with 20% of the effort – but it takes 80% of the effort to complete the project. Just ask any NFL football Team who is in the Red Zone how hard it is to get to the goal.
b) Human complexity
It is my opinion that the human brain, and its deep embeddedness within the human body, involves a level of complexity that is far deeper and more intractable than computer scientists today are willing to admit. As a consequence, it will prove considerably harder to achieve human level intelligence than most practitioners believe. I am reminded of the claims that brain scientists made in mapping the electrical activity of the brain that they had discovered that as much as 90% of that activity was simply “noise.” Later findings determined that the “noise” reflects a very deep level of organizational processing -‐ all of which is essential. Similar claims were made by geneticists who labeled much of the human genome as “junk” DNA – only to find later that the material between the genes acts in a controlling function in what is now known as epigenetics – the process of turning on and turning off genes in the extremely complex choreography required to build proteins matching the needs of the organism. What humans do and what “smart” computers do is quite different. For example, in the game of chess, a computer approaches the game largely as a computational problem, calculating the position values of potential future board configurations in determining the optimal value for its next move – and it is capable of remembering and comparing these configurations to every game of chess that was ever recorded. The human competitor does not have the benefit of such a computational capability or memory capacity and instead draws upon intuition, strategic pattern-‐recognition, and experience in selecting a move. Notably, in the famous match where Deep Blue beat Gary Kasparov, Kasparov was unnerved by an unusual move the computer made in the first game, and claimed that he sensed a human presence. The Deep Blue programmers later were said to confirm that the move was a result of a bug in the program – one that caused Deep Blue to select a move other than the one calculated as optimal. That Kasparov detected such a non-‐routine move and attributed it to human agency is an example of a deep sophistication of the human mind. Some observers also believe that the move by Deep Blue unnerved Kasparov sufficiently that his subsequent play was undermined – leading ultimately to his defeat. Another example of the differences between human and machine intelligence is found in a common story about object recognition. Computer scientists designed a program to recognize images of cats, and trained it with many thousands of photographs. In testing
Page 11 the program, the researchers found that it was quite accurate, but it did make errors. One error was that it identified a photo of two mugs (with facing handles) as a cat – an error that no human 4-‐year old would ever make, since the child knows what a cat is – and what a mug is. The computer does not. Based on what has so far been achieved, my belief is that reverse engineering, modeling, mimicking or duplicating the brain is impossible at this time. There is too much we do not yet know.
c) Hard limits in math and science While resource limitations and the complexity of the human brain are barriers likely to slow down the progress towards artificial human intelligence, there are certain intractable barriers, what I would refer to as the Achilles heel of AI, in the fields of mathematics and physics itself. These include complexity, incompleteness, infinity and quantum physics. For example, consider the simple challenge known as the traveling salesman problem (“TSP”). As a salesman, or a tourist, if you want to plan an optimal route through a few cities, you can make a few calculations of how the mileages all add up between the locations, and pick a route that minimizes the miles. However, as you add to the number of cities, the complexity of the required calculations goes up incredibly fast – much faster than exponentially! Remarkably, in the case of 100 cities, there are more route options (each of which needs to be calculated to solve the problem precisely) than there are atoms in the known universe. Such calculations are not just difficult – they are impossible. The TSP is an example of a broad class of hard problems, some of which show up in fields such as logistics, the manufacture of microchips and DNA-‐sequencing, to name just a few. These problems are referred to in mathematics “NP-‐complete”2. These are very complex problems, contrasted with the more congenial class of problems known as “P”3. P problems can, at least in theory, be solved in a reasonable amount of time. It is unknown whether there is any shortcut to any NP-‐complete problem that would reduce it to a P problem. Quite surprisingly, however, there is a proof that if such a shortcut exists, then all NP-‐complete problems have such shortcuts. The stakes for solving this problem are very high, and a Clay Prize of $1M is being offered for anyone that can solve it. If a P
2 NP refers to problems that are “nondeterministic in polynomial time”, referring to difficulty of knowing the length of time an algorithm will need to run in order to solve it. NP-complete problems are a special subclass, those into which any NP problem can be translated in polynomial time. 3 P problems can be solved by an algorithm in polynomial time – these may be exponentially difficult but are not as hard as the TSP appears to be.
Page 12 shortcut exists for an NP-‐complete problem, the challenges of complexity will be reduced from “impossible” to “very, very hard.” Incompleteness is a problem in formal logic, the foundation for all mathematics, which was discovered in the 20th century by Kurt Gödel. Without belaboring the technicalities, Gödel proved that any logical system (including something as basic as arithmetic) could be either complete (able to prove all “truths”) or consistent (you can’t prove a contradiction), but not both. Thus, in order to avoid inconsistency, the bane of logic, one must sacrifice completeness. This means that there are true statements that cannot be proved! Incompleteness is a somewhat arcane issue, but it does imply that there are going to be things (truths or facts) that we cannot know with certainty. Alan Turing, another brilliant 20th century logician and inventor of the Turing Test discussed above, tackled a somewhat more practical concern, that of constructing a machine (or computational algorithm) that can solve problems in a finite amount of time. Among these problems is the “halting problem”, encountered in the context of an algorithm to determine whether a computational algorithm has completed its task. Turing proved the existence of a universal computing machine, but unfortunately also proved that some problems, e.g. the halting problem, are not amenable to finite solutions – any universal computing machine so constructed would in many cases be unable to complete its task in a finite amount of time. It is an NP-‐hard problem. The fact that mathematics provides the tools by which science can so completely describe the physical world is a mystery. Physicist John Wheeler in 1960 referred to this as “the unreasonable effectiveness of mathematics”. However, it also means that science, including computer science, is subject to the rules and limitations of mathematics. As we have seen above, this means there are problems that cannot be computed, facts that cannot be known, and algorithms that will never stop. Moreover, based on the science of quantum mechanics, there are also physical limits to the hardware on which computing is based. The invention of transistors and semi-‐conductors, and the subsequent miniaturization of computing devices, drove the progress of Moore’s law over the past decades. But such miniaturization cannot progress forever, as the ability to build smaller devices will ultimately founder on quantum indeterminacy. At very small scales, particle behaviors are no longer deterministic and hence cannot provide a platform for digital calculation. There are theoretical discussions about “quantum computing” that may provide interesting new approaches for addressing certain computational problems. However, this is simply a method of tapping into the
The Liar’s Paradox – a key milestone on the path towards Godel’s
incompleteness theorem: “This sentence is false.”
Page 13 lowest possible limits of space. The bottom line is that below this limit, information disappears into the quantum foam that defines physical reality below the Planck scale4.
d) The problem of consciousness One feature of human intelligence, discussed in Chapter II, is that of consciousness, specifically self-‐consciousness. As stated: “There is a mystery in the apparent emergence of self-‐referential awareness from an electrochemical organ (the brain). There is also a paradox since this emergence of self is fundamentally based on a co-‐dependency with other self-‐aware beings.” Despite what some researchers have said in the fields of neuroscience, physics or psychology, the nature of self-‐consciousness and its relationship with human intentionality and free will remains a deep mystery. Some say consciousness is simply an emergent phenomenon of brain states – but that begs the question of what it is that emerges, and what causes it to emerge. Some also say that free will is an illusion, and our actions are determined by our brain states and they flow from strictly physical causes, absent any non-‐physical agency. But in both theories, the emphasis on physical brain states ignores the potent reality of our subjective field of experience, including perception, emotion, intuition and cognition, as well as free will. The question of intentionality, purpose and consciousness also reaches into the very foundations of quantum physics. According to some physicists, without consciousness, there is no driving purpose for the quantum behaviors that create the physical world. The universe can have no reality without consciousness. At the moment, these questions remain metaphysical, or perhaps even theological. I suspect that as the fields of brain science and psychology advance, they will encounter similar hard limits – or perhaps even the same ones -‐ as in math and physics.
4 Quantum foam is a phrase coined by John Wheeler in 1955. The Plank scale (to paraphrase Wikipedia) is the energy scale, named after Max Planck, at which quantum effects of gravity become strong and present descriptions and theories of sub-‐atomic particle interactions break down and become inadequate.
Page 14
6. Conclusion The discussion of super-‐intelligence and its dangers and difficulties, may all seem incredibly esoteric. But the questions are incredibly important. How will we go forward in the technology race, and how can we, technology’s creators, control and direct that process towards ends that serve the human race? The worst imaginable outcome is that we let the process of technological innovation control us. Through apathy, ignorance or complicity, we run the risk of ceding authority over our future to an inexorable machine, whether that machine comes in the form of a super-‐intelligence or merely a thoughtless rush towards novelty and distraction. In Chapter IV, we will address this question, returning to the practical digital technologies that we are dealing with every day. By developing a strategy for dealing with the day-‐to-‐day, perhaps we can illuminate a path for dealing effectively with the longer term and more existential threats of the digital transformation.
Page 15 Selected Bibliography: • Nick Bostrom: Super-‐intelligence – Paths, Dangers, Strategies. 2014 • Noson S. Yanofsky: The Outer Limits of Reason – What Science, Mathematics and
Logic Cannot Tell Us. 2013 • Jonathan Haidt: The Righteous Mind – Why Good People are Divided by Politics and
Religion 2012 • Edward O. Wilson: The Social Conquest of Earth. 2012 • Nicolas Carr: The Shallows – What the Internet is Doing to Our Brains. 2011 • Stephen Pinker: The Better Angels of our Nature, 2011 • Kevin Kelly: What Technology Wants, 2010 • Stuart Brown: Play – How It Shapes the Brain, Opens the Imagination and Invigorates
the Soul. 2009 • Richard Wrangham: Catching Fire -‐ How Cooking Made us Human. 2011 • Melanie Mitchell: Complexity – A Guided Tour. 2009 • Rudy Rucker: The Lifebox, The Seashell and the Soul – What Gnarly Computation
Taught Me About Ultimate Reality, the Meaning of Life and How to be Happy. 2005. • Ray Kurzweil: The Singularity is Near – When Humans Transcend Biology. 2005 • Daniel Goleman: Emotional Intelligence – Why It Can Matter More than IQ. 2005 • Jared Diamond: Germs, Guns and Steel – The Fates of Human Societies. 1997. • Douglas R. Hofstadter: Godel, Escher, Bach – an Eternal Golden Braid. 1979 ONLINE: (Numerous references from Wikipedia and related sources)
Nautilus Magazine online -‐ http://m.nautil.us
On Twitter – http://nymag.com/daily/intelligencer/2013/11/how-‐i-‐learned-‐to-‐love-‐twitter.html
On Facebook – http://www.nytimes.com/2014/06/30/technology/facebook-‐tinkers-‐with-‐users-‐emotions-‐in-‐news-‐feed-‐experiment-‐stirring-‐outcry.html?_r=0 http://www.esquire.com/blogs/news/facebook-‐liking-‐stacey-‐woods https://medium.com/@schmutzie/i-‐quit-‐liking-‐things-‐on-‐facebook-‐for-‐two-‐weeks-‐heres-‐how-‐it-‐changed-‐my-‐view-‐of-‐humanity-‐29b5102abace http://www.psychologytoday.com/blog/wired-‐success/201306/do-‐facebook-‐and-‐other-‐social-‐media-‐encourage-‐narcissism http://www.newyorker.com/tech/elements/how-‐facebook-‐makes-‐us-‐unhappy
On Robots: http://www.popsci.com/scitech/article/2009-‐08/evolving-‐robots-‐learn-‐lie-‐hide-‐resources-‐each-‐other
![Improved Photoactivity of Pyroxene Silicates by Cation ... · rock-forming silicates as they werefound in almostevery igne-ous rock.[19–21] Moreover, it is believed that pyroxenes](https://img.pdfslide.net/doc/110x75/5ed92b706714ca7f476946a6/improved-photoactivity-of-pyroxene-silicates-by-cation-rock-forming-silicates.jpg)
