The Magazine from Forschungszentrum Jülich 02|2008

SUPERCOMPUTERS AND SIMULATION

:: PROTEIN ORIGAMI

:: SPINS: TWISTING AND TURNING

:: FAST COMPUTERS FOR BETTER BLOOD PUMPS

The Magazine from Forschungszentrum Jülich 02|2008

RESEARCH in Jülich

Research in Jülich 2 | 20082

RESEARCH in JülichThe Magazine from Forschungszentrum Jülich

A scientist is applying a 3-D simulation to analyse the flow of breath in the human nostril. He is using the pistol-like object in his hand to fire round particles which move with the flow.

Cover illustration: This simulation shows what happens when laser pulses (orange) hit the surface of a solid (green). Amongst other things, they generate a cloud of electrons (yellow-red) that is accelerated forwards.

2 | 2008 Research in Jülich 3

Together with theory and experiment, computer simulations form the third pillar of research work. They enable us to obtain insights and

knowledge that has been previously inaccessible for physical, technical, financial or ethical reasons. Scientists use supercomputers to investigate very dif ferent questions. How are pollutants and trace substances distributed in the atmosphere and how do they influence our climate? Under what conditions can the simplest protein molecules be created from inanimate matter and thus form the building blocks of life? Why does nature make a distinction between a picture and its mirror image? Into what spatial shape are threadlike protein molecules folded in the body? Join the scientists on their journey of discovery and find the answers to these questions in this issue of “Research in Jülich”.

Apart from our own research with and on supercomputers, at Forschungszentrum Jülich we have also set ourselves the task of providing scientists with supercomputer facilities of the highest performance class in the world. You can discover what this means in concrete terms in the article on “Race for World Leadership”. The following articles show how we develop methods and tools in order to provide support for researchers in the various fields of the computational sciences. However, large computing capacities alone are not enough; you also have to know how to use them. The great added value of Jülich supercomputers for research arises from the combination of expertise in simulation science and the diversified scientific environment.

Here at Jülich we depend first and foremost on human brains and not on computer processors, which is why we also play a pioneering role in training German researchers in the simulation sciences. Together with RWTH Aachen University, we operate the “German Research School for Simulation Sciences” (GRS), which offers masters and PhD courses for

outstanding students. Forschungszentrum Jülich and RWTH thus at the same time also represent a model for cooperation between the universities and nonuniversity research organizations. Cooperation with RWTH Aachen University goes far beyond GRS alone. The JülichAachen Research Alliance (JARA) together with its computer simulation section (JARASIM) provides an institutional framework for joint research projects on a longterm basis.

In the Gauss Centre for Supercomputing (GCS), Jülich cooperates closely with the national supercomputing centres in Munich and Stuttgart. The German supercomputer experts therefore speak with one voice on the international stage. GCS thus represents the German interests in the Partnership for Advanced Computing in Europe project (PRACE), which is preparing the basis for a transboundary supercomputing infrastructure in Europe. By focusing the partners’ knowhow and coordinating the application of funds, researchers throughout Europe will in future have access to supercomputing facilities of a top international calibre. We are, moreover, also participating in the Gauss Alliance, which in addition to GCS incorporates the German regional and specialist computing centres. With our commitment to this Alliance, to GCS and PRACE, we are endeavouring to sustain Jülich’s claim to be Europe’s leading centre for supercomputing and simulation sciences which can bear comparison with the best in the world.

Strong Simulation Science

Prof. Dr. Achim Bachem

Chairman of the Board

of Directors

Dr. Sebastian M. Schmidt

Member of the Board

of Directors


14 17

10

FAST COMPUTERS FOR BETTER BLOOD PUMPSScientists use supercomputers to optimize small implantable pumps for stabilizing patients’ circulation. Jülich simulation experts are helping to perform the required calculations fast and effectively.

SPINS: TWISTING AND TURNINGElementary magnets in a thin layer of manganese on tungsten metal are always arranged in an anticlockwise spiral and never in a clockwise one. Jülich scientists have discovered the reason for this and have now set their sights on possible applications of this curious phenomenon.

29

12:: PROTEIN ORIGAMIDepending on how we fold a sheet of paper, we can create a variety of different figures. Threadlike protein molecules are also folded into very different structures. The simulation of these folding processes on Jülich supercomputers can help us to obtain a better understanding of diseases.

16


IN THIS ISSUE

3 Editorial

:: SNAPSHOTS FROM JÜLICH

6 Research at a glance A kaleidoscope of pictures show highlights of research in Jülich ranging from investigations of the air over Beijing during the Olympic Games and work on strained silicon up to and including a unique instrument providing a glimpse into the brain.

:: FOCUS

8 Computer simulations today and tomorrow

10 Turbulent atmosphere Jülich researchers track down air pollutants all over the globe.

12 Protein origami Into what shape are threadlike protein molecules folded in the body? Simulations provide the answer.

14 Computer simulation is an art

16 Spins: twisting and turningWhy nature makes a distinction between a picture and its mirror image in some magnetic materials.

18 Virtual primeval broth

High pressure, hot water and minerals – proteins could have emerged under these conditions on the primeval Earth.

:: HIGHLIGHTS

20 Race for world leadershipA glimpse behind the scenes: how the fastest civilian computer in the world was installed at Forschungs zentrum Jülich.

24 The future of supercomputing at JülichInterview with Prof. Thomas Lippert, director at the Institute for Advanced Simulation and head of the Jülich Supercomputing Centre

26 An inside viewScientists from the JülichAachen Research Alliance immerse themselves in artificial threedimensional worlds – in future also from different locations.

29 Fast computers for better blood pumpsJülich simulation knowhow helps to optimize lifesaving blood pumps.

32 The doors of perceptionThe computing power of supercomputers is increasing continuously, as is the quality of the simulations.

34 Supercomputing newsSimulations that save human life, software for distributed computing, and an elite school for young scientists.


This magazine focuses on Jülich research using supercomputers as a “tool”. However, Jülich scientists also score great successes in other research areas. We present a few snapshots of this work here.

Research at a Glance

STRAINED SILICONAn innovative material from Jülich will make electronic devices smaller, faster and more economical. Strained silicon is distinguished from conventional silicon by its elongated crystal lattice and improved electronic properties. The Jülich team headed by Prof. Siegfried Mantl has been granted funds totalling € 3.2 million from the Federal Research Ministry and from industry as part of the DECISIF collaborative project in order to further the development of applicationoriented technology.

MICROSCOPY ENTERS A NEW DIMENSIONUltrahighresolution electron microscopy has enabled atomic distances to be measured down to a few picometres, as Jülich scientists reported in the highimpact journal “Science”. A picometre is a billionth of a millimetre. The displacement of atoms in this order of magnitude decides on whether many material properties are technically relevant, for example the ability of semiconductors to bear nanoelectronic structures or that of superconductors to conduct current almost lossfree.

LINK TIPwww.fz-juelich.de


SNAPSHOTS FROM JÜLICH

HONOURS FOR THE NOBEL LAUREATEProf. Peter Grünberg, the Nobel laureate from Jülich, was awarded the Grand Cross with Star of the Order of Merit of the Federal Republic of Germany, and the Order of Merit of the Federal State of North RhineWestphalia (NRW). The physicist was personally honoured by the German President, Horst Köhler, and the Prime Minister of NRW, Jürgen Rüttgers. In 1988, Prof. Grünberg discovered giant magnetoresistance (GMR) – which is now used every day in computers. With more than five billion read heads produced to date, statistically there is one GMR sensor for every member of the human race.

OLYMPIC ATMOSPHEREAtmospheric researchers from Jülich were at the starting line for the Olympic Games in Beijing. Prof. Andreas Wahner was involved in a GermanChinese measuring campaign which investigated the air quality in the Chinese capital during the games – ranging from fine dust to pollution with aerosols, nitrogen oxides, ozone and formaldehyde. The evaluation aims to show how air pollution can be effectively combated in future.

RESEARCHERS COME OUT OF THEIR IVORY TOWERS“Nowadays a leading scientist must also be prepared to establish contact with the mass media and to present his research to the general public.” This is the conclusion reached by Prof. Hans Peter Peters from Jülich in a report published in “Science” on a study he headed. On the whole, scientists are more satisfied with journalistic reporting on their work than one would imagine in view of the general prejudice concerning the “ivory tower”. In fact, about 40 percent of the researchers interviewed from the five leading scientific nations find public reporting conducive to their career.

THE FRAGRANCE OF AGARWOODThere is a great demand for illegally felled agarwood. One kilo of the tropical wood fetches up to $ 10,000. The oriental fragrance of its resin is found in lotions and perfumes. Jülich scientists are working on a longterm project in Indonesia to discover whether agarwood of similar quality can be cultivated.

LOOKING CLOSELY AT THE BRAINThe date is set for April 2009. After two years of preparation, a unique largescale instrument will be put into operation at Jülich to map structures and metabolic processes in the brain with as yet unachievable precision. It is a combined system consisting of a magnetic resonance tomograph (MRT) and a positron emission tomograph (PET). The coils of the MRT generate a magnetic field of 9.4 tesla, which is almost 200,000 times as strong as the Earth’s magnetic field.



FOCUS

Computer simulations are used by scientists from all disciplines to obtain new findings. This magazine focuses on four examples: scientists investigate the atmosphere and climate, biologically important substances, basic material properties and also chemical processes that cannot be recreated in the laboratory. In doing so, they profit from the continuously increasing computing capacity of the Jülich supercomputers. In this way, scientists will be able to study more complex processes and structures in future – an advance that is represented symbolically by the diagram showing columns of figures that turn first into the molecules of simple gases and finally into the complete model of a protein.

:: COMPUTER SIMULATIONS TODAY AND TOMORROW


If the air over Europe isn’t clean then this is not necessarily the fault of the Europeans since the wind carries the

exhaust gases from North American cars and factories over to us. Polluted air doesn’t stop at the borders of countries or indeed continents. “Who is therefore responsible for air pollution when and where? Until we can answer this question, it is difficult to develop appropriate countermeasures,” says Dr. Martin Schultz. The expert from Jülich’s Institute of Chemistry and Dynamics of the Geosphere (ICG) continues: “Satellite observations and other measurements give us valuable pointers, but cannot provide data for all locations and dates.”

The only way out of the dilemma is provided by computer simulations which model the path of the pollutant and trace gases in the atmosphere. This sounds

easy, but it isn’t really. The starting point is government reports on how much waste gas is emitted by each country. Researchers must then take into consideration all of the chemical processes in the air during which pollutants are converted or degraded. Furthermore, attention must also be paid to the influence of the wind, solar radiation and temperature.

FROM EUROPE TO ASIAThe team headed by Schultz has there

fore skilfully coupled an existing com puter model for simulating chemical pro cesses in the troposphere – the lowest layer of the atmosphere – to the global climate model ECHAM5. “In this way we are able to show how European, American or Asian emissions influence the distribution of pollutants in the atmosphere,” Schultz explains. Amongst other aspects, the

Turbulent atmosphereAtmospheric pollutants travel around the world. How far they get depends on how fast they are degraded in chemical processes. Jülich scientists track the gases using their supercomputers and thus determine how air pollution influences climate.

Weather balloons provide important data for atmospheric researchers. But these data alone do not enable scientists to monitor the route of atmospheric pollutants.


FOCUS

scientists also investigated carbon monoxide – CO. The atmospheric pollutant CO indirectly influences the concentration of greenhouse gases in the atmosphere and thus the climate. Since CO is only degraded after one to two months, it can be used to track the routes of trace gases. This is sufficient time for an air parcel to travel once around the world. One result of the simulation is that CO from European sources is generally transported in atmospheric layers close to the ground across Asia towards the east, but, above all in winter, the air can also pollute vast expanses of the Arctic.

However, pollutant and trace gases are not only distributed from south to north and from west to east, but also from bottom to top. “In the tropics, air pollutants can reach atmospheric layers at altitudes of more than 16 kilometres which are otherwise isolated from lower layers. This is, for example, true of CFCs, which are largely responsible for the development of the ozone hole above the poles,” explains Dr. Rolf Müller from ICG. With the aid of the Jülich computer model, ClaMS, he simulates chemical processes in the stratosphere. In midlatitudes, this part of the atmosphere begins about ten to twelve kilometres above the surface of the Earth and thus just above the usual cruising altitude of passenger aircraft. ClaMS makes use of data from the weather services at the nodal points

of the virtual lattice that spans the globe. Fed with information on wind direction, temperature and pressure, the computer follows the path of individual stratospheric air parcels separated from each other by a distance of about 100 kilometres. The concentration of trace gases in the respective air parcel changes continuously as a result of the simulated chemical processes.

During the evaluation of satellite data, it recently became apparent for the first time that small quantities of prussic acid (HCN) were occasionally found in the stratosphere above the tropics and CLaMS was able to successfully model the distribution of this trace gas. “The poisonous HCN has no direct effect in these low concentrations. However, if we understand how the HCN gets into the stratosphere, then we will also be able to better understand the transport routes of other trace and pollutant gases,” Müller explains. Under special wind conditions, HCN resulting from forest fires in the tropics can rise up to the lower edge of the stratosphere. The Jülich computer models point to the fact that HCN is mainly carried into the stratosphere by a process of fairly slow and continual mixing – and not, for example, sporadically with the aid of gigantic tropical thunderclouds.

In future, the research group headed by Müller intends to reduce the distance

between the simulated air parcels and to use CLaMS to track the input of trace gases from the tropics into the stratosphere over a period of several years. “We wouldn’t be able to contemplate such a project if we had the same computing power as we did ten years ago,” says Müller.

IMPROVED FORECASTSThe same is true of Schultz’s team:

“We have only recently become able to calculate models that include the relation between air pollution and climate – thanks to the new generation of supercomputers. And we are now expanding our model to include the chemical processes in the stratosphere which we have not been able to consider so far due to a lack of computer power,” adds Schultz. He is certain that this will make climate forecasts even more reliable. His opinion is shared by the other experts on the Intergovernmental Panel on Climate Change – the winner of the 2007 Nobel Peace Prize. With the results of their simulations, the Jülich researchers will once again be able to make a valuable contribution to the next World Climate Report, which is planned for 2013 and will open the eyes of politicians, companies and the general public to the extent and consequences of future climate change. ::

Frank Frick

Simulation with the Jülich ClaMS computer model: the Asian monsoon circulation over the Himalayas can be seen with tro pospheric air at its heart – recognizable by the high CO content (red) – which is rapidly transported upwards.

This simulation shows as an example the way in which emissions of the atmospheric pollutant CO from sources in Europe (green), North America (orange), Southern Asia (blue) and Eastern Asia (yellow) are distributed in the atmosphere.


Whether elongated muscle proteins, Yshaped antibodies or elaborately formed enzymes with

pockets and depressions – proteins have different shapes depending on their functions. Yet all of them are composed of a chain of small building blocks – the amino acids. Their properties and therefore the forces that act between them, combined with the surrounding water, determine the shape of a protein molecule. The dramatic effects an error can have are shown by the example of prion proteins: incorrect folding can cause mad cow disease in cattle, scrapie in sheep and the lethal CreutzfeldtJakob disease in humans.

Experiments are a timeconsuming and expensive method of determining the shape of many protein chains, even when the precise amino acid sequence is known. An easier approach is therefore often employed where the physical properties of components in a row are used to calculate the folding of the chain – ab initio, as the scientists say. However, because a protein is often composed of several hundred amino acids, such calculations are extremely timeconsuming. This is where a supercomputer like JUGENE in Jülich comes into play. And of course, a clever strategy is also required for the simulation, such as that

developed by Prof. Adam Liwo from the University of Gdansk in Poland.

Liwo starts from a “coarsegrained” approach. This means that he does not need to incorporate each individual atom of the protein into the calculations, but initially just the most important interactions of each amino acid. “You can compare this with two strategies used when looking for mushrooms,” Liwo explains. “You can either systematically comb ev ery inch of the ground, including sandy areas and ponds. Or you first look under trees, where you know from experience that mushrooms grow.” The first strategy corresponds to the calculation of all the

Protein origamiDepending on how we fold a sheet of paper, we can create a variety of different figures: a frog, a crane, or an elephant. In the same way, threedimensional structures are formed by folding threadlike protein molecules. In this form, proteins can take over important functions in the body. The simulation of these folding processes on Jülich supercomputers may help us to gain a better understanding of diseases and provide a starting point for new treatments.


FOCUS

atoms. “It may be more precise, but it would take the entire mushroom season,” says Liwo. By limiting the search to the interesting areas, we end up with a full basket of mushrooms much faster – or with an idea of how a protein is formed. Only then does Liwo take a closer look at important areas of the molecule in order to simulate them atom by atom. This isn’t 100 % accurate at the moment but the virtual molecules are starting to resemble nature more and more closely.

The simulations are based on the principle that under natural conditions a protein chain takes the shape requiring the least energy – similar to the way that water collects in a hollow when left to itself instead of flowing over the edges. If the researchers want to calculate the forces that shape the protein, then they first use the fundamental laws of physics to construct a field of forces. The simulation is, so to speak, trained by a few small proteins whose folding sequences and natural shape are known.

When it is fed this information, the supercomputer runs through all possible folding sequences and predicts the shape of unknown proteins. “This has proved successful for chains of more that 300 amino acids to date,” reports Liwo. His plan is to focus on long muscle proteins next, as well as enzyme complexes made up of a number of subunits, each composed of between 200 and 500 amino acids.

FABULOUS COMPUTING POWERLiwo is convinced that this is possible

with a computer as powerful as JUGENE. He runs his simulations in Jülich: “Because that is where the largest multiprocessor computer in Europe is, and Jülich is an excellent research centre.” He also works on supercomputers in the USA and a small computer in his home town of Gdansk. “The more you eat, the hungrier you get,” was his comment with regard to the constantly growing need for computing capacity. “In the 80s, for my PhD, I calculated the shape of a small peptide made up of nine amino acids in a timeconsuming procedure using a Russian RIAD computer – an exercise that my students can solve in two hours today.” And just five years ago, he hadn’t even dreamed of simulating a protein chain with 500 links, Liwo says. Now, he

thinks it will be possible to simulate whole areas of the cellular machinery – in other words, the interaction between a number of proteins. A suitable tool for this purpose could be the European petaflop supercomputer, which Jülich is developing together with its partners within the framework of the PRACE partnership.

JOINT PROJECTSContact between Liwo’s research

group and Forschungszentrum Jülich dates back to a comprehensive initiative in 2004. Outstanding scientists from the new EU member states were offered access to the Jülich computers on the same

Gdansk was rated as excellent by the independent panel of experts for the John von Neumann Institute for Computing (NIC) – the facility that allocates computing time on the Helmholtz Association’s supercomputers. Since then the Jülich supercomputers have been shaping virtual protein chains.

The origami of the protein molecules on supercomputers is in no way an academic gimmick, Liwo quickly adds. Simulations can help us to understand the coagulation of proteins in Alzheimer’s disease and to find a basis for the development of medications. Another aim is to determine the interactions of proteins and genes – a central task, for example, in cancer research. ::

Wiebke Rögener

Computer calculations show how an important part of a human protein is folded. In the first picture on the left of this sequence, the folding procedure has already begun, and the picture on the right shows the final shape of the protein section.

conditions as German researchers. “In this way, we opened our doors to the very good theoretical scientists in Eastern Europe, and they were able to make use of our excellent infrastructure,” explains Dr. Norbert Attig from the Jülich Supercomputing Centre. “All of us in Europe profited from this move.” Large collaborative projects quickly developed.

“The tedious simulations of protein folding were perfect for calculations on the JUBL and JUGENE supercomputers,” recalls Norbert Attig. “On the one hand, they require a lot of computing time, and on the other, they can be easily divided up between a large number of processors.” In other words, a job for JUGENE with more than 65,000 processors. The application from the working group in


1 3

42

Computer simulation is an artThe supercomputer has developed into a universal research instrument: it is, for example, a virtual telescope, test tube and microscope all at the same time. If you have mastered the art of using the supercomputer properly then you will be rewarded with important results and also some breathtaking pictures.

1 Stars move through the universe at different speeds. The examples coloured red in this cluster of stars are slow while the yellow ones are fast.

2 Laser pulses (left) hit the surface of a solid (blue disc) and produce a cloud of electrons (right).


FOCUS

6 9

5

7

8

3 A simulated quarkantiquark pair under the influence of binding forces. As far as we know today, quarks are the most elementary building blocks of nature.

4 Three wells (blue) collect water from the ground. They are fed by groundwater that flows in the direction of the green arrows. This water transports a pollutant (yellowish brown cloud) along with it.

5 The spatial structure of a protein which results from the folding of long protein chains. Incorrect folding occurs, for example, in Alz heimer’s disease.

6 Swirling eddies: the structural details inside a turbulent flow of gas.

7 Such simulations help researchers to understand the mechanical properties of cell membranes. Under the influence of forces, an elastic network forms a complexly folded structure.

8 Technicolor world: the colours illustrate different concentrations of the trace gas ozone in the atmosphere 38 kilometres above the ground.

9 The flow profile along the mixing chamber of a reformer that is used to obtain a hydrogenrich gas from fuels such as diesel or kerosene. This gas can be used by fuel cells to generate electricity.


Prof. Stefan Blügel from IAS recalls how it all began. “My colleague Matthias Bode from Hamburg told me

about a strange result obtained by one of his PhD students involving a magnetizable material that was divided into domains.” This means that there are socalled boundaries within the material where the direction of the magnetization changes abruptly. Domains are areas of identical magnetization within these boundaries.

The direction of the magnetic field is predetermined on an atomic level by the direction of the electron spin. To put it simply, the spin is like a tiny bar magnet which is aligned to the external magnetic field and which itself also generates a magnetic field. Between the domains, the spin directions between the adjacent electrons must therefore be twisted against each other because the direction of the magnetic field changes here.

However, the change of direction does not occur quite as abruptly as it might appear from the outside. In fact, directly adjacent spins are only rotated against each other at a small angle. Only by stringing together a large number of

Spins: twisting and turningNature is fussy. Elementary magnets in a thin layer of manganese on tungsten metal are always arranged in an anticlockwise spiral and never in a clockwise one. An explanation for this was found by a team of scientists from the Institute for Advanced Simulation (IAS) by pushing the Jülich supercomputer JUMP to its limits. The first applications of this curious phenomenon are already in sight.

neighbours does the overall angle of rotation add up in such a way that it is in agreement with the magnetic orientation of the adjacent domains.

FORGOTTEN FORCE REEMERGES“Our colleagues from Hamburg then

noticed that the spins always rotate in the same direction,” says Blügel. The scientists had, in contrast, expected that the spin direction would be randomly distributed – anticlockwise in one transition wall and clockwise in another. After his conversation with Bode, Blügel vaguely recalled that about 50 years ago a Russian and a Japanese scientist had predicted an effect caused by the interaction


FOCUS

between the orbital motion of an electron and its spin. To date experts had assumed that the force of this effect would be too small to have an impact such as that observed in Hamburg. Blügel, however, began to doubt this assumption.

“Two things came together in this way. Bode’s group had a highresolution scanning tunnelling microscope that was able to detect the strange preference for one spin direction and here in Jülich we had the supercomputer which we could use to explore the possible causes,” says Blügel.

With his computer simulations Blügel was indeed able to reconstruct and explain the observations made by the Hamburg team. The simulations showed how the spin direction depends on the properties of the respective material. “The fact that there is a preferred spin direction at all is a consequence of a quantumtheoretical property,” Blügel explains. “Whereas in conventional mathematics three times four always gives the same result as four times three, this permutability does not exist for certain quantumtheoretical values.”

The Jülich team then used simulations to discover the materials in which this effect is particularly strong. The Hamburg

group then established which of these materials is especially easy to handle in reality. Finally, the two groups came down in favour of tungsten coated with a thin layer of manganese. “Everything worked,” says Blügel. The effect was so strong that it did not occur in the domain walls but was clearly measurable in the form of spirals in the manganese layer. “This finally led to a publication in the high impact journal Nature,” Blügel proudly adds.

SPINS AND POLARIZATIONThe Jülich researcher then focused on

possible applications for the phenomenon. “The spirals can be pushed aside by electric current. Their position could therefore be used for storing information.” However, the behaviour of the spirals in incident light is also promising. The electron spins influence the polarization of the light reflected at the manganese surface. Polarized light is used, amongst other things, for 3D projections – holograms – which make it possible for us to immerse ourselves in artificial computergenerated worlds. ::

Axel Tillemans

Institute for Advanced Simulation (IAS)Since 1 January 2008, supercomputing and the simulation sciences have been

united under the roof of IAS, which currently consists of the Jülich Supercomputing Centre and three scientific institutes that use supercomputers in order to investigate the structure and properties of various materials up to and including biological systems. One of these institutes is headed by Prof. Stefan Blügel. He is convinced that: “The speed with which we obtain new findings will in future increasingly depend on the intelligent application of computers. The optimal working atmosphere for such endeavours is created at IAS by uniting specialists from various disciplines at one institute.” It is planned to integrate other working groups, for example from environmental and health research.

Left-hand picture: The red and green arrows represent spins that can be visualized as small elementary magnets. The upper part of the picture shows how the spins are arranged in a thin manganese layer and below it the mirror image that does not exist in nature. Pictures below: Spins (yellow arrows) in different materials.


Charles Darwin, the father of evolutionary theory, put forward the idea that the first protein mole

cules could have originated in “some warm little pond” with all possible ammonium and phosphorus salts, with light, heat and electricity. He was, however, unable to say how the first step could have been taken from inanimate to animate matter.

The first clues were provided in 1953 by the American Stanley Lloyd Miller. When he reconstructed the possible conditions of the primeval atmosphere in a glass flask simple organic molecules were formed – the amino acids. However, the way in which these building blocks were combined to form proteins and the part that might have been played by minerals containing iron and sulphur remained a matter of speculation.

A research team headed by Dominik Marx, head of the Chair of Theoretical Chemistry at Ruhr University Bochum, has now been able to show for the first time on the Jülich supercomputer that under conditions still found today at hot volcanic vents in the deep ocean it would indeed have been possible for amino acids to combine to form protein chains – even without the biological tools normally required for cells to produce proteins.

SIMULATION NOT SPECULATIONThe scenario of the primeval iron and

sulphur world was not simply reconstructed in a chemical laboratory. The “warm little pond” was created virtually on Jülich’s JUBL supercomputer. Two simulated amino acids – simple glycine molecules – only came into contact on the computer. “In our calculations we

proceed from the basic chemical building blocks, that is to say from the electrons and atomic nuclei of the molecules involved,” Marx explains. No experimental data of any sort were used in these simulations. The scientists only fed the computer with information on the properties of the molecular building blocks involved. The computer then calculated the behaviour of the molecules dissolved in water under various conditions. It was found that the higher the simulated pressure and the hotter the virtual water, the more easily a peptide bond was formed between two amino acids – the fundamental process in protein synthesis. The molecules were exposed to a pressure of up to 20 megapascals, which is roughly 200 times the atmospheric pressure at sea level. The temperature was raised to more than 200 degrees

Virtual primeval brothProf. Dominik Marx from the University of Bochum uses the Jülich supercom puters to investigate how the simplest protein molecules could have originated more than four billion years ago – long before there was any life on Earth. High pressure, high temperatures and sulphurcontaining minerals may have played a part in the origins of life.


FOCUS

Celsius. “It would be extremely difficult to perform such experiments in the laboratory in a controlled manner,” Marx explains. Water has quite different properties under such extreme conditions than in a normal state and the amino acids dissolved in it react with each other more readily. When pyrite containing iron and sulphur was furthermore added to the virtual primeval broth, the protein building blocks came together even faster – the mineral acts as a catalyst accelerating the reaction. “We still don’t know exactly how it works. Pyrite is a model for a number of iron and sulphur compounds that might play a part here,” Marx is quick to add. “Nor do we know whether the first protein molecules were actually formed in the same way as in our simulations. However, by using the supercomputer we were able to show that this path is possible.”

VAST COMPUTING TIME REQUIREDAn enormous amount of computing

time was required to simulate the early history of life on Earth. “If we add everything up then more than 2,000 processors in JUBL were working on this for four months,” says Dr. Norbert Attig from the Jülich Supercomputing Centre. Altogether only about 200 research teams are granted access to one of the Jülich supercomputers per year, and only a fraction of them have access to JUGENE.

“In principle, every researcher in Europe could use our computing facilities,” says Attig. “However, we receive five to seven times more applications than the computing time available.” An international panel of experts evaluates the applications and makes recommendations for allocating the supercomputer resources. The evaluation is based on two criteria. Does the project promise to provide new and relevant findings? And can it be efficiently processed on a large supercomputer?

Both of these criteria applied to Marx’s project. “It is quite simple an extremely interesting question,” says Attig. Scientists were first able to explore this issue by using the Jülich Blue Gene computer. The panel of experts for the John von Neumann Institute for Computing (NIC) – the institution that allocates computing time on the supercomputers of the Helmholtz Association – awarded the project the accolade of “Excellence Project 2008” and the first results have been published in the highimpact “Journal of the American Chemical Society”. The impressive findings have also received considerable media coverage.

Dominik Marx hopes that even more powerful computers will soon be available for his simulations. “JUGENE offers us fantastic new opportunities but our model assumptions are unfortunately still very idealized. More powerful computers would permit us to create more realistic models.” He has reason to be optimistic for the future. “The Gauss Centre for Supercomputing – the association of three national supercomputing centres – has recently taken over the coordination of supercomputing activities in Germany and is helping to ensure that Germany will speak with one voice internationally with respect to supercomputing,” he says. Moreover, on a European level, PRACE (Partnership for Advanced Computing in Europe) will further improve the European supercomputing infrastructure. “However, networking existing centres alone is not enough,” Marx is convinced. “We also need sustainable investments in new computers of the highest performance class in order to remain competitive with colleagues in the USA and Japan.” ::

Wiebke Rögener

A hot volcanic vent in the deep ocean. Here we still find conditions today similar to those that prevailed during the Earth’s early history which could have enabled amino acids to combine into protein chains.

The amino acid glycine (left) is activated in hot water at the interface to pyrite (centre) and then combines with another glycine molecule to form a dipeptide (right).

Minerals containing iron and sulphur, such as this pyrite, facilitate the formation of protein chains.


September 2008: an engineer from the manufacturing company hands Hartmut Peters from the Jülich

Supercomputing Centre (JSC) the official installation manual for the Blue Gene/P supercomputer. The 73 pages contain details of how the cabinets the size of a tele phone box housing the computer – the socalled racks – together with their 65,536 processors should be wired up and supplied with air and electricity. A nonevent? By no means – due to the timing. The Jülich Blue Gene/P – known

as JUGENE for short – had already been in operation for more than six months. “We were the first institution in the world to install a computer of this type and size – parts of the instructions hadn’t even been written and other parts were incorrect,” says Peters. Together, the experts from Forschungszentrum Jülich and the computer manufacturer IBM put the finishing touches to the supercomputer JUGENE. Their experience was then incorporated into the official version of the installation manual.

“Nowadays, no manufacturer can afford to first set up a computer of this size on its own premises, to operate it as a reference and then to supply customers

with machines of identical design,” explains the administrator Jutta Docter from JSC. The experts employed by the first purchaser of a new supercomputer generation play a key role. On top of this, a supercomputer must operate in tandem with data memories and a large number of other computers – and at Jülich also with other supercomputers and European computing centres. Some problems first arise in such an environment and cannot be foreseen by the manufacturing companies in their development laboratories. “We discover how to solve these problems in cooperation with the manufacturer,” says Olaf Mextorf, network expert at JSC.

Race for world leadershipBuying a computer is one thing – getting it to work is another. What is all too familiar to many longsuffering PC owners applies all the more when the computer is “super” and is expected to bring home gold in the race for the fastest computer in the world.


HIGHLIGHTS

COMING UP TRUMPSSince computer technology is devel

oping at breakneck speed, the installation of a supercomputer is a race against the clock. The aim is that the users – in the case of Jülich, the scientists – should be able to perform computer simulations using the maximum power currently available. They thus achieve results faster than other computer research teams and can then be the first to publish them, for example in a highimpact journal. Fast supercomputers therefore give scientists a clear competitive edge. But scientific progress is also closely linked to supercomputing capacity in a quite different way. “More computing power also always means a leap in quality. The results of the simulations become more accurate and reliable,” says Prof. Thomas Lippert. The head of JSC continues: “Sometimes we even discover completely new effects.” The supercomputer then opens up new

fields of research. Since the world’s leading scientists want to perform their simulations on the world’s fastest computers, computers of the highest performance class attract the best researchers to Jülich.

As far as computing power is concerned, experts, funders and journalists always keep their eye on the TOP500 List of the fastest computers in the world. This list is compiled every six months and is published each June and November. The list influences the plans of the supercomputer manufacturers and their development partners and leads to additional pressure of time. The situation is similar to that of a worldclass athlete. He must be top fit in time for the Olympic Games otherwise he will be eliminated in the early heats and won’t get a chance to compete for a medal. Nevertheless, the athlete is still in a relatively relaxed situation compared to the computer manufac

turer. A sprinter still has the chance to win a medal at the next Olympic Games. Due to technological progress, however, a supercomputer will always perform worse in the next TOP500 List than in the previous one.

AROUND THE CLOCKIn September 2007, JSC and IBM in

formed the publishers of the TOP500 List of their plans to upgrade the Jülich supercomputer JUGENE. On 1 October the first computer racks arrived in Jülich. In order to be included in the TOP500 List, JUGENE had to be fully installed and its computing power demonstrated within four weeks. This meant that up to six technicians were working day and night. It took them just eleven days to set up and connect the racks. The first tests were run. Meanwhile, experts in Jülich and the USA worked in three shifts to optimize the operating software and to

View of the computer room in Forschungszentrum Jülich. The series of pictures on this page and the following double-page spread document the installation of the JUGENE supercomputer in October 2007.


write additional diagnostic programs. Finally, they attempted to start the Linpack benchmark that is decisive for the TOP500 List in order to measure the computing power.

“Here at JSC we were naturally all dying to get on the list, as were the IBM staff,” Jutta Docter recalls. Her office was opposite the room used by the IBM technicians working on JUGENE. “I was therefore not only excited myself but also felt their tension rising.” After several attempts, the experts finally managed to get the first presentable result just in time. According to the Linpack test, JUGENE achieved a power of more than 167 trillion arithmetic operations per second or 167 teraflop/s. This is an almost unimaginable result. If every person if the world were able to perform one arithmetic operation of addition or multiplication per second, then the combined intellectual power of all humanity would still be working 30,000 times more slowly

than the computer. During the course of 2008, the performance was actually increased to 180 teraflop/s.

FASTEST COMPUTER IN EUROPEDue to this achievement, JUGENE was

able to call itself the “fastest civilian supercomputer in the world” for six months. As expected, computing power ages rapidly in the fastmoving computer world, but nevertheless even in the next TOP500 List in July 2008, JUGENE was still able to maintain a good sixth place and remained the fastest computer in Europe.

Just four short weeks of – fevered – activity in order to install a supercomputer of this size doesn’t seem very much. Indeed, this recordbreaking tempo was only possible because the Jülich computer specialists had already made meticulous preparations well in advance. Strictly speaking, the story began five years ago. At that time, IBM confided to Klaus Wolkersdorfer, head of “High Performance

Systems” in Jülich, that they had plans for a new type of supercomputer. “It was so top secret at the time that at first I couldn’t even talk to my colleagues about it,” Wolkersdorfer recalls. Instead of continuing to use relatively few extremely powerful processors, IBM intended to install an extremely large number of relatively slow processors. Once relieved of all nonessential functions, these processors should be capable of communicating quickly with each other and rapidly accessing their working memory. “Whereas in 2004 a large number of German experts, for example on the Supercomputer Committee of the National Science Council, remained very sceptical about this concept, we regarded it with great interest,” says Thomas Lippert. The head of JSC continues: “It was immediately apparent to us that this was a chance to break down the barriers of the conventional approach with respect to electricity consumption, space requirements and cost.”

Stowed away in 16 racks, the more than 65,000 processors in JUGENE made it the fastest civilian computer in the world at the time of its launch.


HIGHLIGHTS

Therefore in summer 2005 the JSC experts tested a new rack design with 2,048 processors. “It was a smash hit,” says Wolkersdorf laconically. The Jülich researchers had namely discovered that part of the simulations were indeed readily scalable, as it is known in the technical jargon. The more processors the scientists used, the faster the calculations were finished, without the additional data flow impairing the performance. Forschungszentrum Jülich thereupon ordered an additional seven racks which, together with the existing rack, formed the supercomputer JUBL (Jülicher Blue Gene/L) from early 2006.

FULLY BOOKED SEVEN TIMES OVERJust a few months later it was clear

that even JUBL would not be enough to satisfy the scientists’ demand for computing time. “We were fully booked seven times over,” says Docter. Then came the information that IBM was working on a

new Blue Gene generation. In mid 2006, Thomas Lippert and his team therefore began work on financial planning for JUGENE. Specific preparations for its installation in October then started in spring 2007. For example, new cable routes were required for the power supply leading from the external transformer station to the computer room. “We also had to decide on the location of the racks, calculate the resulting load on the 80centimetrehigh false floor, cut out access holes for the air supply at various points, and also lay twelve kilometres of network cables in the floor,” says Hartmut Peters.

After JUGENE had then successfully passed the Linpack test at the end of October this still didn’t mean that it was ready for the simulation scientists. “For example, the supercomputer was still not communicating with the server computers required to store the data or to allow access to the various research teams”,

says Olaf Mextorf. Furthermore, even the first tests and the Linpack test runs had shown that not all of the more than 65,000 processors were functioning as required. Only then were the JSC experts able to thoroughly analyse the faults and eradicate them. Telephone and video conferences with the American technicians and managers were part of the daily routine. Many a spare part was brought to Jülich from the airport at night by taxi so as not to lose any time since researchers from Jülich, Germany and Europe were already impatient to feed the brand new champion computer with their scientific data. The great day finally dawned in February 2008. Jürgen Rüttgers, Prime Minister of the Federal State of North RhineWestphalia, and Thomas Rachel, Par liamentary State Secretary of the Federal Research Ministry, officially pressed the button launching JUGENE for its users. ::

Frank Frick

Pressing the button officially launching JUGENE for its users (from left to right): Achim Bachem, Chairman of the Board of Directors of Forschungszentrum Jülich, Jürgen Rüttgers, Prime Minister of North Rhine-Westphalia, Thomas Rachel, Parlia-mentary State Secretary of the Federal Research Ministry, and Martin Jetter, General Manager and Chairman of the Board of IBM Deutschland GmbH.


Interview with Prof. Thomas Lippert

The future of supercomputing at JülichThe director at the Institute for Advanced Simulation (IAS) and head of the Jülich Supercomputing Centre (JSC) explains what the supercomputers of the next generation will be like and what role Forschungszentrum Jülich will play in the year 2012.

Question: Prof. Lippert, yesterday at a meeting discussing the Jülich supercomputer JUGENE, which has only been in operation for a few months, one of your staff had to leave early. He excused himself by saying: “I’ve got another meeting. It’s about a new supercomputer.” Can you tell us anything else?

Lippert: Actually, we are already involved in the planning for two supercomputers. We are pursuing a dual concept in Jülich. This is based on the realization that supercomputers can be divided into two classes with respect to the range of applications of JSC users. On the one hand, there are the computers for a

few, very large, dataintensive and highly scalable simulations, computers like JUGENE, which we can use to deal with the grand challenges facing society for instance in the fields of energy and the environment or health. And then, on the other hand, there are the flexible systems for somewhat smaller simulations, which nevertheless make great demands on the working memory and the communication network – in future these will be cluster computers made of standard components. We need both computer architectures in order to be able to satisfy, in a costoptimized manner, the requirements of all the scientists who want to use our computing facilities in Jülich.

Question: But doesn’t every researcher want to process his problems on the most powerful supercomputer available? What do we actually need the cluster computer system for?Lippert: In order to reserve the highend supercomputer for a relatively small number of extremely challenging and interesting research projects – about 30 per year – requiring enormous amounts of computing time. In contrast, on the cluster computer system, just to give you


HIGHLIGHTS

a figure, computing time is available each year for about 200 projects. You should also be aware that by no means every simulation is highly scalable, that is to say it runs faster the more processors are available. Anyway, the dual concept has proved to be extraordinarily successful. In addition to JUGENE, JUMP means that we also have a computer available for, so to speak, our breadandbutter work. Which is not to say that this work is not very important, as shown by the large number of publications in highimpact journals.

Question: What do you mean by the term “cluster computer system” and what are your specific plans for it? Lippert: A cluster computer system is basically composed of computers that are equipped with a complete operating system and consist of powerful standard components – connected via a very fast communication network that is linked to the computing nodes by standard interfaces. Not every computer will have its own cabinet. We have been exploring such systems since 2004 and we are convinced that they offer a superior cost/performance ratio. We now intend to construct such a system on the basis

of quadcore processors from Intel that will be known as JuRoPA (Jülich Research on Petaflop Architectures). We have a number of European companies on board including ParTec and Bull, as well as SUN and Intel from the USA.

Question: At the beginning of the interview, you mentioned that you are currently involved in the planning phase for a second supercomputer. Is this a successor to JUGENE?Lippert: We are actually planning to extend JUGENE from at present 16 to at least 72 racks and thus to increase its performance to the one petaflop range, that is to say one trillion arith metic operations per second. In parallel, we naturally want to keep our users with us on the way to more computing power. After all, at the end of the day you must be able to use the system efficiently.

Question: Does that mean that supercomputing is not just about computing power and computing capacity but also about the knowhow of the operators and users? Lippert: The onsite expertise is what really counts. When I train young people today on supercomputers then they will

play a part as future experts in shaping progress in this field for the next 35 years – supercomputers, on the other hand, are replaced every four or five years. And due to the parallelization of the computer world even more expertise will be required in future – this goes as far as to consider how scientific theories should be formulated so that they can be handled by computers.

Question: And finally, what is your vision? What part will Forschungszentrum Jülich play in supercomputing in 2012?Lippert: Jülich will be the leading name in computational science and engineering in Europe and will operate the supercomputing centre as an element in the new European supercomputer infrastructure. Our institution will bear comparison with the centres in the USA. A computer with a performance of about 10 petaflop/s will be installed here which leading European simulation teams will clamour to use for their calculations. We are cooperating closely with industry to develop the computer systems of the future. ::

Interview: Frank Frick

Thomas Lippert: for him, Jülich is the leading name in Europe as far as computational science and engineering is concerned.


Although he is right in the middle of a burning train, Dr. Armin Seyfried keeps his cool. This doesn’t mean

he is particularly stoical or deathdefying. The train isn’t real and the scientist from the Jülich Supercomputing Centre (JSC) is in the middle of a 3D simulation which is projected onto a threebyelevenmetre screen. But the 3D glasses mean that the image doesn’t stay on the wall. You really believe that you are on the train.

“Even today it is still not clear how much heat is released during a fire on a train,” explains Seyfried. An attempt was made in Sweden to settle the issue using an experiment in which an intercity express was set on fire in a tunnel. “That was very expensive. And both the train and the tunnel were a complete writeoff afterwards. The measurements also involved very great uncertainties,” says Seyfried.

With the aid of simulations developed by him and his colleagues, trains will in future be designed in such a way that the fire won’t be able to spread quickly.

Furthermore, it should be possible to reach the emergency exits as quickly as possible irrespective of where the fire breaks out. The sophisticated simulations, which can only be implemented using supercomputers, enable the train to be set on fire again and again in different places, so to speak. The 3D visualization means that the scientists can observe the way the fire spreads close up and from different perspectives.

Meanwhile, conditions are much colder and windier for Prof. Torsten Kuhlen from the Virtual Reality Centre at RWTH Aachen University. He has just

An inside viewMany people find a good illustration helps them to understand difficult problems better. Scientists are no exception. But they get even more insights from computergenerated, threedimensional worlds. Experts from the JülichAachen Research Alliance (JARA) are therefore developing a network of visualization stations so that researchers at different locations can immerse themselves in the same virtual environment.


HIGHLIGHTS

loaded a visualization of the city of New Orleans in the Aachen “cave”, which his team developed together with the University of Louisiana. The “cave” is a small room where images can be projected onto the walls and floor. “As yet, you can‘t actually touch the objects in our cave – unlike the holodeck in the Star Trek universe. But the visual impression is already very good and is getting better with each computer generation,” says Kuhlen.

Instead of watching the simulation on a computer monitor, Kuhlen puts on what looks like sunglasses with several white balls attached to the frame and enters the cave armed with a sort of toy gun also covered in white balls. Like a giant he now strides through New Orleans, skirts skyscrapers and fires the gun.

THIS IS SCIENCE NOT A GAMEObviously, this simulation is not

a computer game but has a serious scientific background. The “gun” replaces the computer mouse and enables the “caveman” to perform certain actions. In the present case, Kuhlen uses the gun

to release particles which move with the simulated flow of wind in New Or leans.

The white balls on the gun enable a socalled tracking system consisting of two cameras to determine the exact position of the gun in threedimensional space. Kuhlen can, for example, release the suspended particles with the utmost precision in front of a skyscraper or in a street between the highrise buildings in order to observe flow conditions in New Orleans. The balls on his 3D glasses keep the system informed about Kuhlen’s position. The images projected onto the walls and floor are then continuously adjusted to the user’s perspective, which changes with his every move.

“An animated film could not hope to reproduce the opportunities provided by such virtual reality,” explains Kuhlen. “In an animation, a decision has to be made in advance about the user’s position. For example, he wouldn’t be able to see what’s happening behind a highrise building from his perspective whereas in the cave he just goes around the build

ing.” Furthermore, in a film you have to decide in advance from where in the town the suspended particles will be released. If they are released everywhere at the same time then you won’t be able to see anything.

It is obvious that the possibilities offered by such virtual reality depend on the computer capacity available. “This is precisely the reason why our colleagues from Aachen contacted us in the first place. Our supercomputer had the necessary computing power,” says Dr. Herwig Zilken from the Jülich Supercomputing Centre.

ENTERING THE 3D WORLD TOGETHERIn the meantime, collaboration be

tween RWTH Aachen University and Forschungszentrum Jülich within the framework of the JARASIM cooperation (see box on p. 28) has expanded far beyond the provision of computing capacity at Jülich. Zilken’s and Kuhlen’s groups are working jointly on developing what is known as ivNet (pronounced “ivy net”). This abbreviation stands for “immersive

Left: Three scientists in front of the three-by-eleven-metre screen at Jülich. Their 3-D glasses give them the optical impression that they are in the middle of a burning railway compartment. Top: Simulated wind flows between the skyscrapers of New Orleans.


visualization network” and refers to visualization stations connected via a network in which users can immerse themselves – as in the cave in Aachen.

“In contrast to the Aachen cave and our big 3D demonstration screen here in Jülich, this system is transportable and can be easily taken to any research institute and installed there,” explains Zilken. The system has all of the functions of the Aachen cave including the 3D glasses, tracking system and computer mouse. Only the projection area is reduced to a 120by90centimetre screen in order to make it easily transportable.

Individual system stations are already operational in Aachen and Jülich. Kuhlen and Zilken are working on the network and the necessary software. The first goal is to save scientists from Aachen and Jülich who are working on a joint project from having to commute between

their institutions every time they want to discuss a particular problem. ivNet allows the two scientists to remain in their own institutes and meet each other in virtual reality. The crucial point is that they are both in exactly the same scenario. If one of them makes a change using the interactive mouse then the other can experience this directly. The ultimate goal is to make what is initially planned for the relatively short stretch between Jülich and Aachen possible across any distance in the whole world.

The ivNet project could therefore spearhead the next revolution in the field of communications and global networking – in a similar manner to another research network that first came into being 15 years ago in research circles – the Internet! ::

Axel Tillemans

JARASIMIn 2007, Forschungszentrum Jülich

and RWTH Aachen University combined their cooperative projects, which had been running for many years, under the umbrella of the JülichAachen Research Alliance (JARA). Not least due to this collaboration, RWTH was chosen as an elite university by the Federal Research Ministry’s excellence initiative. The JARASIM section of this alliance develops computer simulations as the driving force behind scientific progress and combines the expertise available at the Jülich Institute for Advanced Simulation (IAS) and RWTH. “Both research and teaching profit from JARASIM. Without this institutional interlinkage, the German Research School for Simulation Sciences (GRS), which is intended to train the future scientific elite in computer simulations, would be unthinkable,” says Thomas Lippert, director at IAS.

A scientist uses a sort of toy gun to release a stream of suspended particles. A tracking system with two cameras determines the exact position of the gun in three-dimensional space.


If you need a heart transplant then you haven‘t got much time to spare. However, it can take some time before a

suitable donor is found. In such cases doctors often use a blood pump to support a weak heart. The socalled DeBakey pump manufactured by an American company is a titanium device about the size of your finger which has been implanted into about 440 patients so far. In some heart patients, this relief enables the cardiac muscle to recuperate. However, above all the pump helps patients with a severely damaged heart to survive until a heart is available for transplantation. Prof. Marek Behr from RWTH Aachen University is further developing these lifesaving pumps with the aid of the super

computer. Within the framework of the JülichAachen Research Alliance (JARA), he and Prof. Felix Wolf from the Jülich Supercomputing Centre are optimizing the necessary simulation programs in order to improve the pump.

CONFLICTING DEMANDSAs Goethe put it in his Faust “blood is

a very special kind of juice”, which means that an artificial heart must fulfil different criteria than a water pump. The aqueous blood plasma contains various cells as well as blood platelets. About half the volume of blood consists of red blood cells which transport oxygen. This means that the pump has to fulfil very different requirements. “The pump has to be very

Fast computers for better blood pumpsHeart disease is the number one killer in industrialized countries. A small implantable pump provides support for a weak heart. Researchers make use of Jülich supercomputers and simulation knowhow to optimize the flow inside this pump.

HIGHLIGHTS

Marek Behr from the Jülich-Aachen Research Alliance uses computer simula-tions to develop life-saving blood pumps.


then has a toxic effect on the patient and the damaged

blood cells can no longer transport any oxygen. Nor should the blood platelets be damaged or activated if at all possible otherwise blood clots may be easily formed which then block the blood vessels resulting in a lifethreatening thrombosis. However, trial and error cannot be used on patients in order to improve the pump in such a way that it does less damage to the blood it transports. This is where calculations of flow conditions on the computer can come into play. “This is a typical case where simulations can be used. They help when experiments are

impossible, too resourceconsuming or too dangerous,” explains Prof. Felix Wolf from the Jülich Supercomputing Centre.

“In the first phase up to 2001, we developed the simulation to such an extent that we can be sure that it corresponds to the results of real experiments.” This is how Behr describes the early stages of the project. In the second phase of development, they succeeded in revealing where the blood was damaged most while flowing through the pump. “This occurred to a greater extent than previously assumed in the rear section of the pump, in what is known as the diffuser,” Behr explains. These findings have already influenced the design of a new version of the DeBakey pump. Behr continues: “Since 2006 in the third stage of the project, we have been developing mathematical methods and programs in order to optimize the pump’s mode of operation.”

Blood also makes special demands on the simulation. Whereas the flow behaviour of water can be calculated on a normal PC, a highperformance supercomputer such as Jülich’s JUGENE is required to investigate the liquid flow in such a complicated pump and to study the dynamics of the mixture of liquid and

One result of the simulation: the regions of the blood pump shown in red experience the highest pressure during operation, and those in blue the lowest.

small so that it can be implanted into the human body,” explains Behr. “However, in order to provide sufficient support for the heart it must be able to pump several litres of blood in a minute. Strong shearing forces are the result when so much liquid flows through a small pump in such a short time.” There is thus a danger that the impeller – a component inside the pump – may squash the delicate cells and blood platelets because it rotates 10,000 times a minute.

This must be avoided at all costs. If too many red blood cells are damaged then large quantities of the haemoglobin they contain will leak out. This substance


HIGHLIGHTS

particles. For example, it is very timeconsuming to simulate how the blood pump is started up. “This startup phase of about ten revolutions until the pump is rotating smoothly only lasts a fraction of a second. But in order to simulate this phase we need about ten seconds of computing time on the supercomputer,” Behr explains. Behr turned to Felix Wolf for advice on how to speed up the calculations on the supercomputer. In order to distribute the calculations among a large number of processors on the supercomputer the pump is, so to speak, cut up into small pieces. One processor is then responsible for each of these segments. However, the large number of individual calculations have to be put back together again. This is where one of the difficulties lies. “What happens at the edges of the segments is often interesting,” says Felix Wolf. The exchange of data between the processors is often quite decisive here. “If problems arise then it is no good distributing the task between even more processors. The processors then simply exchange more and more messages and sometimes many of them are simply superfluous, as if they were sending empty envelopes.”

TRACKING DOWN WEAK POINTSThis is why the simulation of the flows

in the DeBakey blood pump was initially no faster or more accurate when more than 900 processors were involved. Wolf used special software to get to the bottom of this problem. He incorporated small sensors into the simulation program. That is to say, little pieces of program that record all the details of each mathematical operation. Using the program package developed at Jülich and

known as Scalasca (Scalable Performance Analysis of Large Scale Applications), he analysed these data and thus tracked down the weak points in the simulation program. On the basis of this analysis, it became possible to stem the flood of “empty envelopes”. “The simulation now runs efficiently on more than 4,000 processors and is roughly five times as fast as it was before,” Wolf is pleased to say. In this way, new versions of the pump can be examined much faster in a virtual test run, which means that patients also benefit more quickly. For example, a scaleddown version of the pump has been developed for children aged from five to sixteen years suffering from heart disease. “Such successes are only possible because engineers and computer scientists work hand in hand in JARA,” says Wolf.

He is hoping that the European Partnership for Advanced Computing in Europe (PRACE) initiative will lead to further progress. Individual supercomputers are already available in Europe that are well up in the international ranking. “But we are still not in as good a position as the USA,” says Wolf. However, European plans for petaflop class computers providing 1,000 trillion floating point operations per second are already well advanced. “This will further improve conditions for researchers in Europe who want to perform complex simulations,” Wolf confidently adds. A longterm goal may also be coming a bit closer: blood pumps which do not just keep the patient alive until a suitable donor is found but which can actually provide a permanent replacement for the human heart. ::

Wiebke Rögener

Felix Wolf between the racks of the Jülich supercomputer. Together with his team, he has made the calculations for the blood pump simulations faster and more effective.

31

The design of the doors and thus the “face” of the supercomputers at Jülich has changed over the course of the years. But the computer’s brain behind the doors has changed even more. Its computing power has increased enormously – especially with the present generation of devices. At the same time, the simulations performed on these computers have become more precise, detailed and realistic: demonstrated here by the example of the computational grid for simulating the climate over Europe showing how the meshes have become finer and finer.

The doors of perception

2.6 Gigaflop/s10 Gigaflop/s

614 Gigaflop/s

32 Research in Jülich 2 | 2008

33

HIGHLIGHTS

614 Gigaflop/s

167,000 Gigaflop/s

2 | 2008 Research in Jülich


News from Supercomputing

At the first industry seminar held by the EU PRACE project in September 2008, the foundation for further cooperation was laid by top executives from leading companies in the automobile, chemical and electronics sectors, scientists from 13 European countries, and representatives of the European Commission. “The seminar showed that very positive synergies can be achieved between research and industry in supercomputing and in simulation,” said Jean François Hamelin, head of IT at the French energy utility EDF, after the seminar. In the PRACE project (Partnership for Advanced Computing in Europe), coordinated by Forschungszentrum Jülich, 16 partners are preparing for the establishment of infrastructure which will provide European researchers with access to supercomputer resources of the very highest level.

Powerful partners

In nature, electrical charges play a decisive part in protein folding, catalytic processes on surfaces and many other phenomena. Jülich algorithms tailored to supercomputers help to simulate complex systems of charge carriers in bigger dimensions. Thanks to funding to the tune of € 1.5 million for the ScaFaCoS project promised by the Federal Research Ministry, scientists from Jülich are now cooperating with German partners in further developing efficient programs for physical and biological issues and in making them available to the research community.

Highly charged computers

Experts from the Jülich Supercomputing Centre have made a thorough revision of the European grid software UNICORE and have now published Version 6. Grid computing is a form of distributed computing which enables scientists and engineers to solve difficult problems requiring extensive computing capacity. UNICORE provides users all over the world with safe and intuitive access to computer resources and data, also at supercomputing centres. The software is now also being used by industrial companies and public institutions. www.unicore.eu

Software for distributed computing


HIGHLIGHTS

The German Research School for Simulation Sciences (GRS), founded by Forschungszentrum Jülich and RWTH Aachen University, has opened its doors. Since winter semester 2008/2009, handpicked master’s students are being trained here in the applications and methods of computer simulation for science and engineering. In April 2008, GRS had already welcomed the first physicists and engineers working on their PhDs in Jülich and Aachen. www.grssim.de

Back to school for the scientific elite

In future, computer models will be able to rapidly predict how people attempt to escape from the danger of a fire or terrorist attack in football stadiums, railway stations or public buildings. Developed by Jülich scientists, simulations will form the backbone of a computerassisted evacuation system. It will help to guide people to the best escape routes and to optimally deploy safety personnel and the emergency services. The Federal Research Ministry is providing about € 4.6 million to fund the Hermes project coordinated by the Jülich Supercomputing Centre, and also involving industrial partners, as part of its programme on “Research for Civil Security”.

Simulations save lives

In November 2008, the Federal Government and the federal states of BadenWürttemberg, Bavaria and North RhineWestphalia agreed to provide € 400 million for the stepbystep expansion of the Gauss Centre for Supercomputing (GCS) at its sites in Jülich, Garching and Stuttgart. “We are extremely grateful for this support since it provides us with the unique opportunity of playing a decisive role in shaping the new European infrastructure with petaflop computers,” says Prof. Achim Bachem from Forschungszentrum Jülich on behalf of the board of directors of GCS. www.gausscentre.eu

Decisive engagement

PUBLICATION DETAILS

Research in Jülich Magazine of Forschungszentrum Jülich, ISSN 14358514 Published by: Forschungszentrum Jülich GmbH | 52425 Jülich Conception and editorial work: Dr. Frank Frick, Dr. Angela Lindner (responsible according to German press law), Kosta Schinarakis Authors: Dr. Frank Frick, Dr. Wiebke Rögener, Dr. Axel Tillemanns Design and layout: Graphic Shop, Forschungszentrum Jülich Photo credits: Forschungs zentrum Jülich (cover illustration, pp. 8 – 9, p. 11, p. 14, p. 15 all except Fig. 6, p. 17 left, p. 21, p. 26, p. 28 bottom, pp. 32 – 33, p. 34 top left), R. U. Limbach (p. 3, p. 4 top, bottom right, p. 6, p. 7 top left, bottom centre, bottom right, p. 12, p. 17 top, p. 19 bottom, pp. 22 – 25, p. 31 right, p. 34 top right, p. 35 top right), K. Peters (p. 10 bottom), JARASIM, RWTH Aachen University (p. 2, p. 27, p. 28 top, p. 29 bottom, p. 30 top), creativ collection (p. 10 top, p. 20 top, p. 30 bottom, p. 35 bottom), MARUM, Uni ver sity of Bremen (p. 18 top), micromedcv.com (p. 29 top), University of Hamburg (p. 4 bottom left, p. 16), E. Scheiner; N.N. Nair; D. Marx; Ruhr University of Bochum (p. 19 top), C. Czaplewski; A. Liwo; University of Gdansk (p. 13), Getty Images (p. 7 bottom left), Thomas Klink (p. 7 top right), J. Schumacher (p.15 bottom left), U. Schulzweida & N.P. Noreiks; Max Planck Institute for Meteorology (map of Europe pp. 32 – 33) Translation: Language Services, Forschungszentrum Jülich Contact: Corporate Communications | telephone +49 2461 61 4661 | fax +49 2461 61 4666 | info@fzjuelich.de Printed by: Druckerei Schloemer GmbH Print run: 5,000

Member of:

Documents

The Magazine from Forschungszentrum Jülich 02|2008