Progress in Mathematics Volume 123

Series Editors J. Oesterle A. Weinstein

Lie Theory and Geometry In Honor of Bertram Kostant

Jean-Luc Brylinski Ranee Brylinski Victor Guillemin

Victor Kac Editors

Springer Science+Business Media, LLC

Jean-Luc Brylinski Department of Mathematics Penn State University University Park, PA 16802

Victor Guillemin Department of Mathematics MIT Cambridge, MA 02139

Ranee Brylinski Department of Mathematics Penn State University University Park, PA 16802

Victor Kac Department of Mathematics MIT Cambridge, MA 02139

Library of Congress Cataloging In-Publication Data Lie theory and geometry : in honor of Bertram Kostant / Jean-Luc

Brylinski... [et al.], editors. p. cm. - (Progress in mathematics ; v. 123)

Invited papers, some originated at a symposium held at MIT in May 1993. Includes bibliographical references. ISBN 978-1-4612-6685-3 ISBN 978-1-4612-0261-5 (eBook) DOI 10.1007/978-1-4612-0261-5 1. Lie groups. 2. Geometry. I. Kostant, Bertram.

II. Brylinski, J.-L. (Jean-Luc) III. Series: Progress in mathematics : vol. 123. QA387.L54 1994 94-32297 512'55~dc20 CIP

Printed on acid-free paper © Springer Science+Business Media New York 1994 Originally published by Birkhäuser Boston in 1994 Softcover reprint of the hardcover 1st edition 1994

Copyright is not claimed for works of U.S. Government employees. Al l rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopy­ing, recording, or otherwise, without prior permission of the copyright owner.

Permission to photocopy for internal or personal use of specific clients is granted by Springer Science+Business Media, LLC, for libraries and other users registered with the Copyright Clearance Center (CCC), provided that the base fee of $6.00 per copy, plus $0.20 per page is paid directly to CCC, 21 Congress Street, Salem, MA 01970, U.S.A. Special requests should be addressed directly to Springer Science+Business Media, LLC.

ISBN 978-1-4612-6685-3

Text prepared by TeXniques, Inc. from disks.

9 8 7 6 5 4 3 2 1

Bertram Kostant

Table of Contents

Preface . . . . . ix

Normality of Some Nilpotent Varieties and Cohomology of Line Bundles on the Cotangent Bundle of the Flag Variety Bram Broer

Holomorphic Quantization and Unitary Representations of the Teichmiiller Group

1

Jean-Luc Brylinski and Dennis McLaughlin ........... 21

Differential Operators on Conical Lagrangian Manifolds Ranee Brylinski and Bertram Kostant .............. 65

Groups and the Buckyball Fan R. K. Chung, Bertram Kostant, and Shlomo Sternberg . .... 97

Spinor and Oscillator Representations of Quantum Groups Jintai Ding and Igor B. Frenkel. . . . . . . . . . .

Familles coherentes sur les groupes de Lie semi-simples et restriction aux sous-groupes compacts maximaux Michel Duflo and Michele Vergne . . . . . . . . .

The Differential Geometry of Fedosov's Quantization Claudio Emmrich and Alan Weinstein . ...

Closedness of Star Products and Cohomologies Moshe Flato and Daniel Sternheimer ....

The Algebra of Chern-Simons Classes, the Poisson Bracket on it and the Action of the Gauge Group

127

167

217

241

Israel M. Gelfand and Mikhail M. Smirnov . . . . . . . . .. 261

A Distinguished Family of Unitary Representations for the Exceptional Groups of Real Rank = 4 Benedict H. Gross and Nolan R. Wallach ....

Reduced Phase Spaces and Riemann-Roch Victor Guillemin . . . . . . . . . . .

289

305

viii Table of Contents

The Invariants of Degree up to 6 of all n-ary m-ics Roger Howe ................ .

Tensor Products of Modules for a Vertex Operator Algebra and Vertex Tensor Categories Yi-Zhi Huang and James Lepowsky .....

Enveloping Algebras: Problems Old and New Anthony Joseph . ............ .

Integrable Highest Weight Modules Over Affine Superalgebras and Number Theory

335

349

385

Victor G. Kac and Minoru Wakimoto . . . . . . . . . . . 415

Quasi-Equivariant V-Modules, Equivariant Derived Category, and Representations of Reductive Lie Groups Masaki Kashiwara and Wilfried Schmid ...... .

Meromorphic Monoidal Structures David Kazhdan ........ .

The Nil Hecke Ring and Singularity of Schubert Varieties Shrawan Kumar

Some Classical and Quantum Algebras Bong H. Lian and Gregg J. Zuckerman

Total Positivity in Reductive Groups George Lusztig . . . . . . . . . .

The Spectrum of Certain Invariant Differential Operators Associated to a Hermitian Symmetric Space Siddhartha Sahi. . . . . . . . . . . . . . . . . . .

Compact Subvarieties in Flag Domains Joseph A. Wolf . ......... .

457

489

497

509

531

569

577

The Work of Bertram Kostant

Over a period of more than four decades Kostant has produced some extremely creative, important and influential mathematics. It continues to be very striking to all of us that, among his many papers, virtually all are pioneering with deep consequences, many an article spawning a whole field of activity. While there have been many papers that either further develop or generalize papers by him, few have improved on his treatment of any topic. Most of Kostant's work is related to Lie theory and its many facets, including algebraic groups and invariant theory, the geome­try of homogeneous spaces, representation theory, geometric quantization and symplectic geometry, Lie algebra cohomology, Hamiltonian mechanics, modular forms, etc ... In fact, one striking consequence of Kostant's body of work is that Lie theory (and symmetry in general) now occupies a larger role in mathematics than it did before him. Kostant's depth of range ex­tends to major papers in algebra and mathematical physics, and includes a number of unpublished papers which have also been very influential.

Much of Kostant's mathematics demonstrates the value and the im­portance of achieving the necessary level of generality at the same time as new ground is broken. By the same token, however, his work illustrates his own belief that there are certain objects and structures in mathematics which are so beautiful and so fundamental that they justify many years of effort to unearth their hidden treasures.

The present volume is a tribute to Bert Kostant. The 22 high level articles assembled here reflect the diversity of his influence on many fields. While the excellence of the articles will speak for themselves, we will at­tempt to give an overview of Kostant's work to date (early July, 1994). We have chosen to divide the exposition according to the decades of his scien­tific activity, as this seems to lead to a more lively picture of the ensemble of his work. We have attempted to at least illustrate the variety of the papers, their origin, their influence on mathematics (and also on physics); we have not tried to discuss the works of other mathematicians except to the extent that they are directly related to or have influenced Kostant's work and his thinking. No doubt much has been omitted and many con­nections have escaped our notice. We hope that this overview can serve as a guide, as experience shows again and again that so much is gained from reading the original source of a discovery, and Kostant's papers are very well-written.

There was a Symposium in honor of Kostant at MIT in May, 1993. Some (but hardly all) of the papers in this volume grew out of talks

x Preface

given there. The Symposium, entitled "Lie Theory, Algebra and Geomet­ric Quantization", seems to be a fitting descriptive synthesis of Kostant's works.

This summary was prepared by the Editors. We thank David Vogan for giving us his account of Kostant's work in representation theory up to the mid 1970s. Any inaccuracies are of course the responsibility of the Editors.

50s

Some of Kostant's first papers ([1] [4] [5]) are devoted to a study of the holonomy groups of homogeneous spaces. Kostant described the Lie al­gebra of the holonomy group in terms of certain operators Ax associated with Killing fields X. In [4] and [5] the holonomy group is described for a large class of homogeneous spaces. In [10] Kostant gives a characteri­zation of those affine connections which admit locally a transitive group of connection preserving transformations. These results are reproduced in the treatise of Kobayashi and Nomizu. In [3] it is proved that for any Rie­mannian metric on the sphere sn, the holonomy group is always the full rotation group. This basic result is derived from a representation-theoretic fact proved in [10], namely, that if a subgroup G of SO(n) (for n ~ 5) has no invariants in I\i jRn for all 0 < i < n, then G = SO(n).

Kostant gave in [2] a complete classification of real Cartan subalgebras of a real simple Lie algebra. The classification is done in terms of suitable families of orthogonal roots. In the second part of [2] there are complete tables for all the Cartan subalgebras. This second part was widely circu­lated, but remained unpublished as the PNAS complained that the tables were too complicated! The first (and independent) publication of the lists was in a paper by Sugiura (the lists of Kostant and Sugiura coincide).

In [8] Kostant gave his famous formula for the multiplicity of a weight J.L

in a finite dimensional representation V,x of a simple Lie algebra g. The mul­tiplicity is the alternating sum over w in the Weyl group of the expression S;P(w· (,X + p) - (J.L + p», where s;P denotes the Kostant partition function, which counts the number of expressions of a weight as a sum of posi­tive roots. The Kostant multiplicity formula is a precursor to the theory of Verma modules and to the Bernstein-Gelfand-Gelfand resolution of a finite-dimensional representation. It served as a model for many subse­quent character formulas (for instance, Blattner's formula). The paper [8] also contains some other curious identities, which deserve to be studied (see the review of [8] by Freudenthal). There is much work of Kostant on generalizations of [8] which has remained unpublished - notably he has a formula for the restriction of a representation to an arbitrary reductive

Preface xi

subgroup, and another result deals with a subalgebra which is the fixed point algebra of some involution.

In [7] Kostant gave a beautiful infinitesimal characterization of the standard representations of the classical groups; they are characterized by the fact that the Lie algebra representation contains operators of rank 1 or 2. Combined with his unpublished results mentioned above, this led him to prove that the compact pairs (SU(n), SU(n -1)), and (SO(n), SO(n -1)) are the only ones such that any irreducible representation of the big algebra restricts to a multiplicity free representation of the small algebra.

In the 50s Kostant wrote the first [9] of a series of fundamental papers on Lie theory which have changed the subject. In [9] Kostant studied the principal three-dimensional subalgebra (TDS) .6 (which was introduced by Dynkin and by Freudenthal) of a complex simple Lie algebra g, and described the structure of g as a representation of .6 under the adjoint action. This .6-module structure is related to the Betti numbers of the corresponding Lie group G and to the heights of the roots of g. Kostant was thus able to give a theoretical proof of an empirical observation of Shapiro and Steinberg, relating these two sets of quantities. More precisely, g decomposes under .6 as the direct sum of representations of dimension 2mi + 1, where the mi's are the exponents of g. Here the exponents are defined in such a way that ili (1 + t 2mi +l) is the Poincare polynomial of G. Equivalently, if (e, h, J) is the standard basis of the Lie algebra .6, the mi

are the eigenvalues of h on ge. The principal TDS is nowadays a standard tool in Lie theory.

Another important new object introduced in [9] is a distinguished con­jugacy class A of regular elements of order equal to the Coxeter number h; the elements of this class restrict to the Coxeter element, which is the product of the simple reflections in the Weyl group, on a suitable Cartan subalgebra. The exponents of G are described from the adjoint action of the Coxeter element on g. Kostant gives a uniform theoretical proof of the equality hl = 2r where h is the Coxeter number, I is the rank, and r is the number of positive roots; this equality was observed empirically by Cole­man. The conjugacy class A turns out to be of fundamental importance in a variety of domains, as evidenced by the work of Kostant in the 70s. It is proved in [9] that any regular element of G has order at least h, and any element of G of order exactly h belongs to the Coxeter class. The paper [9] was presented by Koszul in Bourbaki Seminar no. 191.

In the 50s Kostant was also actively working on Lie algebra cohomol­ogy theory. While studying the G-invariants in f\P g* which correspond to primitive cohomology classes in H 2P(BG), Kostant found a strange identity which he related to old results of Frobenius on the representation theory

xii Preface

of the alternating group [6]. This identity turned out to be the Amitsur­Levitski theorem, which says that the the skew-symmetrized 2n-fold prod­uct of 2n matrices of size (n, n) is O. Kostant also gave in [6] a new identity for skew-symmetric matrices. The paper [6] was the subject of Bourbaki Seminar no. 243 by Dieudonne.

It is probably in the late 50s that Kostant proved the important theo­rem that any orbit of a unipotent algebraic group acting on an affine alge­braic variety is closed. The proof can be found in M. Rosenlicht "On quo­tient varieties and the affine embed dings of certain homogeneous spaces", Trans AMS 101 (1961), 211-223; cf. also D. Birkes, Ann. of Math. 93 (1971), 459-475, and Rosenlicht's book Algebraic Groups.

60s

In 1961 Kostant published the second in his series of major papers on Lie theory [11]. The paper [11] establishes a firm link between representation theory and Lie algebra cohomology. For any parabolic subalgebra p of g, with nilpotent radical m and Levi subalgebra gl, Kostant determines the action of gl on the Lie algebra cohomology HP(m, VA»' The representations of gl occur with multiplicity one, and the representations 1Tu which do occur are parametrized by a E WI, where WI c W is a subset of the Weyl group. This implies both a generalization of the Weyl character formula and its interpretation in terms of Lie algebra cohomology. Lie algebra cohomology was to play an ever increasing role in representation theory. Lie algebra cohomology is fundamental in the algebraic representation theory of semisimple Lie algebras, starting with Vogan's algebraic classification theory for Harish-Chandra modules. It is central as well in the localization theory of U(g)-modules (where the fibers of V-modules are Lie algebra cohomology groups). The Hecht-Schmid character formula is expressed in terms of Lie algebra cohomology, very much in the spirit of [11]. The formula of Kostant for Hi(m,VA ) made its appearance in the theory of automorphic forms and of locally symmetric spaces in Zucker's paper on L2-cohomology and intersection cohomology; it is now an important tool in this field. The method of proof of Kostant's formula is very interesting, as it uses methods similar to Hodge's in Lie algebra cohomology. Finally, [11] has been generalized by Garland and Lepowsky to Kac-Moody Lie algebras; see their article in Invent. Math. 34 (1976) for this generalization and its applications to the Macdonald-Kac formula. A version of Lie algebra cohomology in the affine Lie algebra case (mostly in degree 0) is used in constructing the conformal blocks in the WZW (Wess-Zumino-Witten) conformal field theory, and in the construction of tensor categories due to

Preface xiii

Kazhdan and Lusztigj see the article by Kazhdan in this volume for more recent developments. It is also a crucial standard tool in gauge theories and in string theory, namely BRS theory, on which Kostant and Sternberg worked in the 80s.

The paper [16], which is a continuation of [11], solves the mystery dimHi(n) = dimH2i(GIB), where n is the nilpotent radical of b. This equality had been proved by Bott using topological methods and the Riemann-Roch theorem, and the question arose to find a direct proof. Kostant introduced a new kind of laplacian with the help of which he set up an isomorphism between [Hi(m)®Hi(m)*)]91 and H2i(GIP). Both are parametrized by WI j moreover, Kostant proved the remarkable fact that the class in H2i (G I P) of the Schubert cycle Zn associated to any a E WI corresponds under this isomorphism to the canonical (identity) tensor in 7rn ® 7r; C Hi(m) ® Hi(m)*. This generalizes a classical formula relating the Schubert calculus with multiplication of Schur functions. In particular, Kostant was able to determine which harmonic polynomials on the Cartan subalgebra ~ correspond to Schubert cycles in G I B, under the isomorphism between harmonic polynomials and H* (G I B).

The paper [15] (announced in [14]) is another milestone in Lie theory and geometry, and its many fundamental results are of constant use. One main construction is that of the graded space H of harmonics inside the symmetric algebra of a semisimple Lie algebra g, as the space annihilated by all positive degree invariant constant coefficient differential operators. One of the main results in [15] is the tensor splitting 8(g) = I ® H, where I is the algebra of invariants (which is a polynomial algebra by Cheval­ley's theorem). [15] also gives the analogous splitting for U(g). Kostant proved in [15] that H is the space spanned by all powers xk of nilpotent elements of g. Under the action of g, H decomposes with finite multipli­cities, and each irreducible finite-dimensional representation V occurs with multiplicity equal to the dimension of its zero-weight space. Kostant used sophisticated commutative algebra to prove that the nilcone 91 in 9 (the cone of nilpotent elements) is a normal variety. An important point here is the use of the cross-section e + gf for the set of regular (Le., maximal dimensional) adjoint orbits, where (e, h, J) is the basis of a principal TDS. The normality is an extremely important result. It implies that all regular functions on the unique orbit Oe of regular nilpotents extend to regular functions on 91 so that R(Oe) = R(91). Moreover it implies that the sym­metric algebra 8(g) maps onto the algebra R(O) of regular functions for each regular orbit 0, and that R(O) identifies with H as a representation of g. The surjectivity implies easily that the global sections of the sheaf V G / B of differential operators over GIB is a quotient of U(g), a fact first

xiv Preface

noted by J.-L. Brylinski and Kashiwara, which is important to their so­lution of the Kazhdan-Lusztig conjecture and of course also to the more general Beilinson-Bernstein localization theory.

Kostant also established that the G-orbits in !Jt (the so-called "nilpo­tent orbits") are finite in number. Kostant's results inspired much further work on the geometry of nilpotent orbits, due to Barbasch, Borho, Broer, J.-L. Brylinski, R. Brylinski, DeConcini, Ginsburg, Graham, Guillemin, Hesselink, Hinich, Joseph, Kraft, Levasseur, Lusztig, MacPherson, McGov­ern, D. Peterson, Procesi, Richardson, Slodowy, Smith, Stafford, Sternberg, Vogan, Wolf and many others. The normality of nilpotent orbits was proved for sl(n) by Kraft and Procesi, Closures of conjugacy classes are normal, Invent. Math. 53 (1979), 227-247, but for other types there are non-normal nilpotent orbits.

The well-known Springer correspondence between nilpotent orbits and representations of the Weyl group is strongly connected to Kostant's work in [151. Furthermore the moment map T*G / B - 9 defines Springer's resolution of singularities T*G / B - !Jt. It follows using again the normality of!Jt that R(!Jt) = R(T*G / B).

Building on [91, Kostant introduced in [15] the notion of generalized ex­ponents for a finite-dimensional representation V as the eigenvalues Ai (V) of h on the space vge

• By [9], for the adjoint representation one recovers the usual exponents of g. The generalized exponents of V are equal (with the correct multiplicities) to the degrees in which V occurs in H. Hesselink and D. Peterson (independently) proved that the generating function of the generalized exponents is, up to a simple shift, equal to the q-analog of O-weight multiplicity, where the latter is defined by substituting a natural q-analog of Kostant's partition function into Kostant's weight multiplicity formula. There is a wealth of beautiful results on generalized exponents and on the q-analogs of weight multiplicities, due to Hesselink, D. Peter­son, Lusztig, Kato, R. Brylinski (some of them under her former name R. Gupta) and others.

Kostant's work on functions on regular semisimple and nilpotent or­bits and generalized exponents was extended to spaces of "twisted" func­tions (sections of homogeneous line bundles) by R. Brylinski in two articles (in JAMS 2 (1989) and in the Dixmier Festschrift, Birkhiiuser PM 92). There the orbits are considered as varieties fibered over G / B. The reg­ular semisimple orbits are realized as, in Kostant's terminology, shifted cotangent bundles, while of course 0 e C T* G / B is open. Corresponding to each irreducible finite-dimensional g-representation VI' with non-zero zero-weight space Brylinski constructs a natural ideal II' of R(!Jt) with the property that the multiplicity of any VA in II' is equal to the dimension

Preface xv

of the weight space V,:; there is a more general construction for general V,. using the simply-connected cover of De. Broer subsequently studied in detail these ideals, proved Brylinski's conjecture on the generation of the ideals, and obtained strong results on the vanishing of cohomology groups of line bundles on T*G / B- see Invent. Math. 113 (1993) and his paper in this volume. Furthermore Broer obtained striking results on the subregular orbit very much in the spirit of [15).

The paper [15) was presented by Godement in Bourbaki Seminar no. 260.

The splitting theorem of Kostant was extended to quantized envelop­ing algebras by A. Joseph and G. Letzter. Reference to this work and a discussion of various problems related to Kostant's work can be found in Joseph's paper in this volume.

The paper [29) with S. Rallis (announced in [23) [24)) generalizes many of the results of [15) to symmetric pairs (g, t). Let G be the complex adjoint group, and let Ko C G be the subgroup fixed by the Cartan involution B. They study the action of the complex group Ko on the complexified vector space Pc, and show that the algebra of Ko-invariant polynomial functions on Pc is a polynomial algebra in I homogeneous generators Pi>"" Pl' For each A E C l , we have a closed algebraic subvariety Pc,'\ defined by the ideal generated by the Pi - Ai' The algebraic group Ko acts algebraically on each PC,'\. For general A, this is a smooth subvariety of PC,,\, which is a single Ko-orbit. For arbitrary A, Kostant and Rallis study the structure of each Pc,'\, and show that it decomposes into finitely many Ko-orbits; there is exactly one closed Ko-orbit, consisting of semisimple elements, and one open orbit (which is not always connected). The variety Pc,o is the nilcone, and consists of the nilpotent elements in Pc. Kostant and Rallis prove an analog of the Jacobson-Morozov theorem, which says that any nilpotent e E Pc is contained in a simple three-dimensional subalgebra which is B­stable. Unlike the situation in [15), here the nilcone may have several irreducible components and these components may be non-normal. The paper [29) also gives the generalization to symmetric pairs of the tensor splitting theorem of [15).

The papers [15) and [29) have greatly influenced the development of "effective" invariant theory due to Vinberg, Popov, Kac and others. It turns out that if one replaces the involution by an arbitrary automorphism of finite order a of a complex simple Lie algebra gc, the representation of the fixed Lie algebra gc on each eigenspace of a has remarkable properties similar to those established for involutions in [29). Moreover these examples almost exhaust all finite-dimensional representations V of simple Lie groups with nice invariant-theoretic properties (such as freeness of the algebra the

xvi Preface

algebra of polynomial invariants, finiteness of the number of orbits in each subvariety VA of V).

Furthermore [15] and [29] gave a strong impetus for further work in invariant theory of reductive group actions in the sense of Hermann Weyl. In addition to the mathematicians mentioned above particularly relevant here is the work of Brion, Carrell, Dixmier, Howe, Knop, Littleman, Luna, Vust and many others.

Early in the 60s, Kostant started to develop a major theory called geometric quantization, for the purpose of geometrically constructing uni­tary representations of Lie groups. Kostant was influenced by a number of geometric constructions of representations done in the 50s and the early 60s, to wit:

- the Borel-Weil-Bott theorem, which in particular constructs finite­dimensional representations of a complex reductive group as the space of holomorphic sections of a line bundle over the flag manifold G / B;

- the Bargmann-Segal construction of the Stone-von Neumann repre­sentation of the Heisenberg group;

- the constructions of the principal series representations by 1. Gelfand, M. Graev and S. Gindikin, and the construction of the discrete series by Harish-Chandra;

- Chevalley's construction of the Spin representation; - Kirillov's complete description of the unitary dual of any unipotent

group, in terms of the coadjoint orbits in his paper "Unitary representations of nilpotent groups", Uspekhi Math. Nauk 17, 1962.

Soon after Kirillov's paper, Kostant saw that all these constructions could be fitted into a unique theory, and he conceived of his geometric quan­tization program which is summarized in the note [19]. An aspect of the theory which is crucial for applications to representation theory is the obser­vation that coadjoint orbits carry a canonical symplectic form. The skew­symmetric pairing on each tangent space to the orbit was written down in Kirillov's 1962 paper; this is an algebraic construction. The geometric fact is that the resulting 2-form on the coadjoint orbit is a symplectic form, i.e., a closed non-degenerate 2-form. This geometric construction was also part of the program developed independently by Souriau. The idea that for "integral orbits" the symplectic form is the curvature of a line bundle was conceived by Kostant near that time, as were the ideas of prequantization, the topological notion of integrality and its relation with induced represen­tations. Kostant lectured on all this already at MIT in 1964-65. Iwahori's own notes from these lectures were circulated widely both in the US and abroad and were referenced in the book Representations des Groupes de Lie Resolubles by Bernat, Conze, Duflo, Levy-Nahas, Rais, Renouard and

Preface xvii

Vergne. Kostant's handwritten lecture notes from this course and from subsequent courses at MIT in the 1960's also circulated widely and had enormous impact on the development of the field from 1966 onward. In fact a copy still resides in the Mathematics Library of the University of Paris 7. Additional lectures by Kostant in this period were written up and cited often: Kostant's note [19) and his Phillips Lectures at Haverford in 1965 (the notes taken by Husemoller from these lectures will appear soon in book form).

Kostant also introduced at that time the notion of polarization in its general form. This will be taken up later, because the important paper [28) appears in 1970, but we will illustrate here one significant way in which Kostant's approach differs from a more algebraic one, which was developed by Bernat, Conze, Dixmier, Dufio, Pukanszky, Rais, Vergne and others (cf. Dixmier's book Enveloping Algebras, the book Let;ons sur les Representations des Groupes by L. Pukanszky, and the aforementioned book by Bernat, et al.). For an algebraist, the main ingredient of the orbit method is that to each coadjoint orbit G . >. c g*, there should be attached a Lie subalgebra q containing gA, which is maximal isotropic with respect to the skew-form (this is the algebraic notion of polarization). Then >. gives a Lie algebra character of q, and if one can integrate it to a char­acter X of the corresponding Lie group Q, then the relevant representation of G is obtained by inducing X from Q to G. For Kostant, the notion of polarization means more generally an involutory lagrangian distribution of complex tangent spaces on the coadjoint orbit, and the vector space of the representation is obtained as the space of sections of the line bundle that are annihilated by the vector fields which belong to the distribution. A G-invariant lagrangian distribution of tangent spaces on G· >. ~ GIGA amounts to a GA-invariant subspace q of g containing gA, which is maxi­mally isotropic with respect to the symplectic form. The lagrangian dis­tribution is integrable if and only if q is a Lie subalgebra of g; thus one recovers the algebraic notion of polarization as a special case.

In 1970 Kostant's seminal paper [28) appeared, to be followed by [30) [33) [37). The geometric quantization program of Souriau, which has many common points with that of Kostant, is discussed in his article in Comm. Math. Phys. 1 (1966), and his book Structure des Systemes Dynamiques, published in 1970. Souriau's book will soon be translated into English. We will have more to say later about Souriau's work as we present the paper [28). Kirillov developed his own program in the 60's: his paper in Punct. Anal. Appl. 1, no. 4 (1968) gives the geometric construction of the symplectic form on a coadjoint orbit; the article of Kirillov in Funct. Anal. Appl. 2 (1968) gives for nilpotent groups his famous formula expressing

xviii Preface

the character of a unitary representation as the Fourier transform of the Liouville measure on a coadjoint orbit. Kirillov' s book Elements of the Theory of Representations was published in Russian by Nauka in 1972 and in English by Springer-Verlag in 1976.

Kostant became very interested in Hopf algebras in the middle 60s, un­der the influence of the paper by Milnor and Moore on the classification of graded-commutative and cocommutative Hopf algebras over algebraically closed fields of characteristic zero. Kostant proved an extension of the Milnor-Moore theorem, stating that any cocommutative Hopf algebra over an algebraically closed field of characteristic zero is the smash-product of a group algebra and the enveloping algebra of a Lie algebra. This theorem and its proof were published by Moss Sweedler, who was Kostant's student, in his well-known book Hopf Algebras. This structure theorem was later to playa role in Kostant's definition of a super Lie group.

The paper [20] introduces Kostant's Z-form Uz of the enveloping alge­bra of a semisimple Lie algebra. The construction uses the Chevalley basis, and in fact is in some sense dual to Chevalley's construction of algebraic groups over Z, nowadays called the Chevalleygroup schemes over Z. It involves divided powers, which previously were used mostly in algebraic topology, but which were to be a cornerstone of the theory of crystalline cohomology, developed by Grothendieck, Berthelot and Illusie in the late 60s. The theory of quantum groups over Z, which has recently been devel­oped by Lusztig, Kazhdan, S. Gelfand, DeConcini-Kac, Procesi, Andersen­Jantzen-Soergel, and others, can be viewed to some extent as a q-analog of the Kostant Z-form. The study of differential operators over flag vari­eties in positive characteristic, which is in its early stages, uses the Kostant Z-form as its starting point.

The papers [12] and [13] give some major results on differential forms and cohomology. A theorem in [12] is that the Hochschild homology of a regular algebra A over a field of characteristic zero identifies with the regu­lar differential forms over A (in the sense of Grothendieck). This beautiful result left open the question of constructing the exterior differential on dif­ferential forms in the context of Hochschild homology. This problem was to be solved by A. Connes around 1983, with his introduction of the operator B and of cyclic cohomology, and dually by Tsygan and by Loday-Quillen, in the context of cyclic homology (which is in fact closer to the work [12]).

In [13], Hochschild and Kostant prove that the ordinary cohomology of an affine algebraic group is always computed by the cohomology of the complex of regular differential forms. This was mentioned by Grothendieck as a motivation for his famous theorem that the cohomology of any smooth complex algebraic variety is the hypercohomology of the de Rham complex

Preface xix

of sheaves with respect to the Zariski topology.

70s

In the 70s Kostant continued to make important strides in representation theory and to develop geometric quantization. Kostant's program on geo­metric quantization and the geometric construction of representations was developed in the address [27] to the IeM in Nice. The paper [28] (which is really a treatise on line bundles and prequantization) was published in 1970 but in fact represents the theory that Kostant had developed since the early 60s. Kostant developed systematically the theory of line bun­dles with connection (L, '\7) over a manifold, and emphasized that the pair (L, '\7) should be considered together. He presents a cohomological descrip­tion of the group of isomorphism classes of such pairs (L, '\7), using Cech cohomology. The sort of Cech cohomology that he considered corresponds to the complex of sheaves ~x ~A~, where ~x is the sheaf of smooth C* -valued functions on X, and A~ is the sheaf of smooth complex-valued I-forms. This complex of sheaves is the smooth analog of a complex in­troduced by Deligne a few years later to describe holomorphic line bundles with connection. Kostant then obtains two main general results on line bundles with connections:

- a complex closed 2-form w appears as the curvature of some connec­tion on some line bundle if and only if it satisfies the quantization condition: the cohomology class of 2~i is integral;

- if w satisfies the quantization condition, then the isomorphism elasses of pairs (L, '\7) with curvature w form a principal homogeneous space under the group Hl(X,C*). Such a pair (L, '\7) is called a quantum line bundle.

Kostant then considers a symplectic manifold (X,w) and studies the problem of lifting a group G of symplectomorphisms to a group of diffeo­morphisms of the quantum line bundle L, which preserve the connection. This is the geometric stage of the prequantization method. Kostant first examines the infinitesimal problem, and finds (for X simply-connected) that there is a central extension of the Lie algebra H am(X) of Hamilto­nian vector fields, which lifts to a Lie algebra of vector fields on the line bundle. He then shows that the problem of lifting the action of g = Lie( G) ean be deeomposed in two parts:

(1) first the action of g should be Hamiltonian; (2) then the central extension of II am(X), pulled-back to g, should

split. Analyzing (1), Kostant is led to the concept of a moment map, and he

xx Preface

proved that the action is Hamiltonian if and only there exists a moment map for the central extension of g. He showed that for 9 finite-dimensional and semisimple, (1) and (2) automatically hold.

These results are also found in the book Structure des Systemes Dyna­miques of J.-M. Souriau. However Kostant went further by showing that the obstruction to the problem of lifting the group G to a group of diffeo­morphisms of L is controlled by a central extension G of G. This beautiful result and the construction were found again in later years by many other authors, few of whom attribute it to [28].

The last step of the prequantization program consists in giving a canon­ical action of G on the space of sections of L. This representation will be unitary if the manifold is compact and the pair (L, V') has a hermitian struc­ture. There is very beautiful formula for the infinitesimal action, which is given in [28] and in the book of J.-M. Souriau; for the case of a cotangent bundle, this formula specializes to a formula of I. Segal.

For the construction of representations of G in the spirit of the orbit method the relevant symplectic manifolds are the coadjoint orbits and their coverings. Kostant shows in [28] that these are exactly all the homogeneous symplectic manifolds for which the group action is Hamiltonian. Kostant then analyzes connections on a homogeneous line bundle over a coadjoint orbit G . A. Such a homogeneous line bundle corresponds to a character X of the stabilizer GA. The existence of such a group character implies an integrality condition on the corresponding Lie algebra character, which Kostant relates to his quantization condition on the curvature of a line bundle.

To achieve true quantization, there remains the question of dividing by two by the number of variables on the orbit. For that purpose, one needs a polarization in the sense of Kostant, meaning a distribution of lagrangian subspaces on the symplectic manifold X. The quantization space should then, according to [26] [27] [28] [33], be the space of sections of L which are killed by any V' e, where the vector field e belongs to the distribution; this geometric idea achieves the halving of the variables. An example of a polarization is given by the tangent spaces to the fibers of a fibration X ---? M whose fibers are lagrangian. Such a lagrangian fibration is necessarily a shifted cotangent bundle in the sense of Kostant. In fact, Kostant more generally considers an involutory distribution V of complex lagrangian subspaces inside the complexified tangent bundle; it is necessary here to assume that V + V is also involutory. For instance, a Kaehler manifold admits such a complex polarization, and this already occurs in the Borel-Weil-Bott theorem, as well as non-positive complex polarizations. The advantage of considering several types of polarizations is illustrated for

P'reface xxi

instance by work of Jeffrey-Weitsman and others on the moduli spaces of representations of the fundamental group of a surface.

The paper [3~] with Auslander was a major achievement of geometric quantization, in which a complete description of the unitary dual of a solv­able Lie group of type I was obtained. The ideas of geometric quantization were really necessary there, since there are non simply-connected orbits, for which, by Kostant's theory, there will be several associated quantum line bundles. Furthermore Auslander and Kostant solved the problem of characterizing the solvable groups which are of type I: G is of type I if and only any A E g* is integral and its orbit is locally closed in g*.

In [33] and [37] Kostant refines geometric quantization by the intro­duction of the half-form bundles. Kostant is thus able to explain the shift by ! . fi between the actual eigenvalues of the harmonic oscillator and those predicted by the WKB method. This "metaplectic correction" in geometric quantization is described in Chapter 10 of Woodhouse's book Geometric Q'uantization, Oxford Univ. Press, 1991. In [33] and [37] Kostant further developed the symplectic spinor bundle and applied it to geometric quan­tization. As symplectic spinors contain various half-form bundles, they can be used to define, pair and differentiate half-forms. Kostant's ideas were developed further by Guillemin and Sternberg. They form a very ele­gant framework for the kernels appearing in H6rmander's Fourier integral operators and for the Maslov index.

Although the constructions of the Hilbert space in geometric quantiza­tion require the choice of a polarization, part of the philosophy of Kostant is that, at least in the good cases which occur in representation theory, the Hilbert space should be independent of the polarization. In the mid 70s Kostant found a mechanism for actually proving in many cases this independence of polarization. The isomorphism (which makes crucial use of the half-form bundles) is obtained by means of a pairing called the BKS pairing (after Blattner-Kostant-Sternberg), and which is defined geometri­cally. A nice example of a BKS pairing, which is already present in [3~], gives the standard isomorphism between the Fock representation and the Heisenberg representation. The book Geometric Asymptotics by Guillemin and Sternberg contains a discussion of joint work by Blattner, Kostant and Sternberg.

Holomorphic methods of geometric quantization are discussed in the article of J-L. Brylinski and McLaughlin in this volume. The papers by Emmrich-Weinstein and by Flato-Sternheimer discuss another approach to quantization, usually called deformation quantization.

It is very interesting to see that connections between symplectic geom­etry (more precisely, Poisson brackets) and secondary characteristic classes

xxii Preface

are investigated in the article by Gelfand and Smirnov in this volume. Motivated by the study of moment mappings undertaken in [28],

Kostant described the moment map for the adjoint action of a maximal torus T on an adjoint orbit 0 of a compact Lie group G (note that there is a G-equivariant isomorphism between 9 and g*, so an adjoint orbit may be viewed as a coadjoint orbit). The moment map 7r is then given by the projection of an element of 0 to its component in t. Kostant proved his famous convexity theorem, which says that the image of 7r is the convex hull of the finite set 0 n t. For the case of SU(n), this recovers a theorem of Horne on the diagonal part of hermitian matrices.

This convexity theorem was generalized in the 80s by Atiyah-Bott and by Guillemin-Sternberg to any Hamiltonian action of a torus on a compact symplectic manifold. Atiyah and Bott showed that the theorem can also be deduced from the work of Duistermaat and Heckman on asymptotic ex­pansions of oscillatory integrals. Guillemin and Sternberg gave interesting asymptotic results on the repartition of weights in a pencil Vn 'A of finite­dimensional representations, which converges weakly to the push-forward to t ~ t* by the moment map of the Liouville measure of the coadjoint orbit of A. This is a sort of quantitative measure-theoretic version of the Kostant convexity theorem.

Kac and D. Peterson showed that the Kostant convexity theorem still holds true for coadjoint orbits in Kac-Moody groups, and recent work of Bloch, Flaschka and Ratiu shows that the theorem extends to the group of area preserving diffeomorphisms of an annulus.

The paper [32] contains a second (more sophisticated) convexity the­orem; this is roughly speaking a non-linear analog of the first theorem, in which the linear projection t ~ t is replaced by the projection G ~ A to the factor A in the Iwasawa decomposition G = KAN. This second theo­rem is much harder to prove; symplectic proofs for it have been obtained only in the complex case by Weinstein and by Ginzburg. Recent work of Lu and Ratiu suggests that this theorem should be analyzed in the setting of the "quantum category", using q-groups, q-manifolds, etc ....

A third theorem in [32] is a beautiful statement about the 3 sides of a geodesic triangle in a symmetric space of negative curvature, which says that if u is one side of the triangle (viewed as an element of p), and if v is the vector sum of the other two sides, then v lies in the convex hull of the W-orbit of u. This recovers the Goldsen-Thompson inequality tr eX+Y :$ tr eXeY for x and y hermitian matrices.

In the mid 70s Marsden-Weinstein used the moment map in their im­portant theory of Hamiltonian reduction. The connection of Marsden­Weinstein reduction with geometric quantization was studied by Guillemin

Preface xxiii

and Sternberg in the 80s. The article by Guillemin in this volume describes the relation of Marsden-Weinstein reduction with the Riemann-Roch the­orem.

Another major preoccupation of Kostant in the 70s is the application of Lie theory to the integrability of Hamiltonian systems. In [45] Kazhdan, Kostant and Sternberg studied the Calogero system (which is described by the potential E~j (qi - qj)-2). This system had been shown to be completely integrable by rather complicated techniques. It is shown in [45] that the Calogero system can be obtained by Hamiltonian reduction from a linear Hamiltonian system, which not only easily implies complete integrability, but also enabled them to write down the trajectories in more or less closed form. One very intriguing aspect of [45] is that it contradicts the then accepted wisdom that reduction simplifies the mechanics, since the opposite is true for the Calogero system. The same feature was later found by Atiyah and Bott to apply to the moduli space of flat bundles on a compact oriented surface, which is obtained by Hamiltonian reduction from the space of connections. Another important feature of [45] is a connection between Howe pairs and Marsden-Weinstein reduction.

In [46][47] Kostant derived the complete integrability of the Toda sys­tem (which is described by the potential Ei eq,-qi-l), from a general result which is known nowadays as the Kostant-Symes lemma. In fact, Kostant described a generalized Toda type system associated to any simple Lie al­gebra, with potential Ei eO" the sum running over the simple roots (the classical Toda system corresponds to .G[(n». Kostant identifies the phase space of the Toda system with a set of Jacobi matrices, which in turn identifies with a coadjoint orbit of a Borel subalgebra, to which he applies the Kostant-Symes theorem. To state this important criterion, let g be a finite-dimensional Lie algebra, and let a and b be Lie subalgebras such that g = a EI1 b as vector spaces. Then the Kostant-Symes lemma states

Theorem. Let oX E g* be a linear form which vanishes on [a, a] and on [b, b]. For any function f on g*, denote by 1>. the function on a* given by f>..(l) = f(l + oX) for l E a*. Then if f and g are G-invariant functions on g*, the corresponding functions 1>.. and g>.. on a* Poisson commute (equiv­alently,their restrictions to any coadjoint orbit in a* Poisson commute).

In the application, a is a Borel subalgebra and b is the Lie algebra of a maximal compact subgroup. Kostant then derives the complete integra­bility of the Toda system and integrates it explicitly in terms of the matrix coefficients of the fundamental representations of g.

The Kostant-Symes theorem was applied by Adler, van Moerberke, 1.

xxiv Preface

Gelfand and others to show the complete integrability of many Hamiltonian systems.

Kostant was a pioneer in the 70s in the theory of supermanifolds and of super Lie groups. His theory is exposed in [41]. Motivated by Berezin's (et ai.) work and by Kac's classification of the simple Lie superalgebras, the theory was built up on Kostant's work in the 60s on cocommutative Hopf algebras. The Lecture Notes [41] gives the analogs in the supercate­gory of the Frobenius theorem, of the tangent and cotangent spaces, of de Rham theorem and homogeneous spaces. Symplectic structures (of even type) are also studied, and the super analogs of Hamiltonian mechanics, coadjoint orbits (of even linear functionals) and quantum line bundles are also treated. A very nice feature is that the Cartan-Chevalley construction of the Clifford algebra from the exterior algebra now appeared as a case of geometric quantization in the super category. In this fashion the Clifford algebra appears as an "odd" version of the algebra of differential operators on a manifold. This analogy was to be pushed further by Witten and Get­zler in their local proof of the Atiyah-Singer index theorem. It is interesting to note that the theory of BV quantization involves odd symplectic forms.

Kostant's theory of supermanifolds remains one of the most important approaches to the field; other theories were developed later by B. DeWitt and also by Manin for algebraic supermanifolds. Needless to say, superman­ifolds and super Lie groups have been of great importance in theoretical physics since the 1970s.

The paper [34] was motivated by a question by 1. Segal; it explains why the space of solutions / of the wave equation 6./ = 0 on compactified Minkowski space is conformally invariant, even though the operator 6. is not invariant under the conformal group SO(4,2). Kostant introduced in [34] the notion of a quasi-invariant differential operator on a flag manifold and classified all these operators. They are parametrized by singular vec­tors in a certain Verma module, and each operator defines a subrepresenta­tion of a principal series representation. [34] should be further studied, and should have applications to differential operators on automorphic forms.

During the period where Kostant was making huge strides in geomet­ric quantization, symplectic geometry and Hamiltonian systems, he was also extremely active in pioneering new methods in representation theory. Kostant undertook his famous work on spherical principal series [35], for which he was awarded the Steele Prize of the AMS for an influential pa­per in 1990. The citation, in part, reads "he used algebraic methods to solve completely a problem that has resisted analytic methods, and as a consequence found a simple and powerful construction for new series of representations". We will describe informally the results in [35] using the

Preface xxv

language of Harish-Chandra modules. A Harish-Chandra module X for a group G is called spherical if the space XK of K-invariant vectors is non­zero. If X is spherical and irreducible, then X K is one-dimensional. The spherical Harish-Chandra modules X(..\) are parametrized by a character ..\ of a maximally split abelian subalgebra n. It was known since the 1950s that the X(..\) are unitary for..\ purely imaginary, and Bruhat showed in his thesis that X(..\) is irreducible for ..\ regular (which means that ..\ has trivial stabilizer in the so-called little Weyl group W). However the essential gen­eral problem to determine when X(..\) is irreducible was open; this is the problem that was completely solved in [35]. In particular Kostant showed that X(..\) is irreducible whenever ..\ is purely imaginary, and more gener­ally when ..\ is in an explicit critical strip. He also showed that the unique irreducible quotient Z("\) of X(..\) admits an invariant Hermitian form if and only if there exists some w E W such that w· ..\ = -'X. Kostant's description of the hermitian form is quite explicit and enables one to prove its positivity by analytic continuation.

The methods of Kostant in [35] grew out of the theory of harmonics in [29]. For each K-type 'Y occurring in the harmonics, he constructs a polynomial valued matrix p-r, and was able to compute its determinant. This led to Shapovalov's work on Verma modules. The irreducibility and unitarity problems were expressed in terms of the p-r. A further wonderful property of these matrices is that the multiplicity of a K-type in an irre­ducible spherical representation is the rank of a certain matrix R-r which is a product of two matrices of type p-r.

This work of Kostant has had many applications to the theory of sym­metric spaces. It led Helgason to the precise description of the values of ..\ for which the so-called Poisson transform, from a space of hyperfunction sections of a certain line bundle on G / P, to a space of functions on G / K, is injective. This was important in the proof of the Helgason conjecture, due to Helgason in the rank one case, and to Kashiwara, Minemura, Okamoto, Oshima and Tanaka in the general case.

Another ground-breaking paper from this period is [36], where Kostant studies the infinitesimal characters which can occur in the tensor product of an infinite-dimensional representation with one of finite dimension. This was the starting point for the theory of translation functors, which was developed very successfully

- by Bernstein-Gelfand-Gelfand and by Jantzen in the category 0; - by Knapp, Stein and Zuckerman for Harish-Chandra modules; - by Borho and Jantzen for primitive ideals. These translation techniques have become a fundamental technique in

representation theory. In their localization theory Beilinson and Bernstein

xxvi Preface

show that the geometric counterpart of a translation functor is the opera­tion of twisting a V~-module over the flag variety by a line bundle.

In the paper [44] Kostant solves the question of which representations of a semisimple Lie group G admit a Whittaker model (for a given character X of a maximal nilpotent subalgebra n), or equivalently which representations occur in the Whittaker model, which is the representation of g induced from that character. If V is the dual of an irreducible Harish-Chandra module, then V admits a Whittaker model if and only if its annihilator is a minimal primitive ideal; in that case, the space of Whittaker vectors has dimension equal to the order of the Weyl group. This has applications in the theory of automorphic functions. The paper [44] contains a number of other impor­tant results. For instance, it is shown that higher n-cohomology vanishes for representations which admit a Whittaker model. [44] gives structure theorems for the structure of the n-invariants of U(b) and a splitting the­orem for the enveloping algebra of b.

Around that time Kostant obtained results on the cascade of orthog­onal roots and the polynomial ring structure of the center of U(n). These results are referred to in the paper by A. Joseph in Jour. of Algebra, 48 (1977).

In the early 70s, following up on his work with Rallis, Kostant came up with his correspondence between nilpotent elements e in Pc and nilpotent elements z E g. The correspondence is as follows: Given e E Pc which is nilpotent, it was proved in [29] that there is a TDS Oc of gc, containing e and stable under (J. Kostant further observed that, after conjugating e by an element of Kc if necessary, one may assume that the TDS is stable under complex conjugation. Thus we obtain a real TDS 0 of g. Then take z to be a nilpotent element of o. Conversely, given a nilpotent z E g, by Jacobson-Morozov z is contained in a real T DB 0 C g. After conjugating o by an element of G if necessary, we may assume that 0 is stabilized by (J. Then e is taken to be a nilpotent element of Oc n Pc. Kostant was able to show that in nearly all, but not in all cases, these give two inverse bijections between Kc-conjugacy classes of nilpotent elements in Pc and G-conjugacy classes of nilpotent elements in g. Kostant's work was not published, but he gave numerous lectures on this correspondence in the early 70s. Several years later Sekiguchi proved the correspondence in all cases; he also established the correspondence in a more general setting in­volving two commuting involutions of a real semisimple Lie algebra; see Sekiguchi's paper in Jour. Math. Soc. Japan. 39 (1987). This is why the correspondence is widely referred to as the Kostant-Bekiguchi correspon­dence. The correspondence is a crucial tool in important recent work of Schmid and Vilonen on the characteristic cycles and associated varieties of

Preface xxvii

Harish-Chandra modules. Another unpublished result of Kostant in this period is the determi­

nation of the outer derivations of the nilradical n of a Borel algebra; this result is quoted and generalized in G. Leger and E. Luks "Cohomology of the nilradical of a Borel subalgebra of a semisimple Lie algebra H*(n, n)", Trans. AMS, 195 (1974).

In [39] Kostant gave a beautiful representation-theoretic version of the MacDonald-Kac formula from the early 1970s, which expresses 1Jdim (o) as a sum over the weight lattice. His formula involves the Coxeter conjugacy class a which was so important already in [9]. To write it down, let P+ be the set of dominant weights; for >. E P+, let X.\ denote the character of the corresponding finite-dimensional representation V.\, and let c(>.) be the eigenvalue of the normalized Casimir operator. Then we have:

1]dim(g) = E n(a)· dim(V.\). qC(.\) •

.\EP+

This and other results of [39] were presented in Bourbaki Seminar no. 483 by Demazure. For related "super" developments, see the paper of Kac and Wakimoto in this volume. Some results of [39] are used by Howe in his article appearing in this volume.

Fegan and Millman recognized that the above formula for 1Jdim(o) is really a statement about the heat kernel on K; see their paper in Amer. Math. Monthly 93 (1986).

80s

Kostant continued in the 80s to work on geometric quantization. The paper [50] contains the notion of shifted cotangent bundle, which is very impor­tant in geometric quantization as a symplectic manifold X equipped with a smooth submersion X -+ M with lagrangian fibres is necessarily a shifted cotangent bundle of M. Kostant also develops in [50] a new, sharpened, symbol calculus which allows him to recover the correct coadjoint orbit (as opposed to a nilpotent orbit) attached to an ideal in the enveloping alge­bra. These results of Kostant and related results are also discussed in W. Borho-J.-L. Brylinski, Bull. Soc. Math. Fr. (1990).

The 80s saw many applications of geometric quantization to the rep­resentation theory of reductive Lie groups. Many of these applications originated in the coherent cohomology of holomorphic vector bundles over some open domains in flag bundles, and were motivated by the geometric quantization program of Kostant and by the Langlands conjecture. These

xxviii Preface

applications were developed by Schmid and by Wolf already in the 60s and 70s, and there were important works in the 70s and 80s by them and by Guillemin, Rawnsley, Sternberg and many others. To a large extent, the theory of analytic localization of representations, developed in the 80s by Hecht, Milicic, Schmid and Taylor, is very close to geometric quantization. We will refer the reader to the articles by Kashiwara-Schmid and by Wolf in this volume for further information.

In the papers [60J [64J Kostant gave a beautiful geometric construction of the minimal representation of SOC 4,4). This representation H is realized as the kernel of a Laplace type operator acting on the space of sections of some line bundle over S3 x S3. Kostant constructs a hermitian pairing on H via a Radon transform which is closely related to triality. He gives a beautiful description of the restriction of H to the maximal compact subgroupK = SU(2)xSU(2)xSU(2)xSU(2): they form a pencil. Kostant proves that the annihilator of H in the enveloping algebra is the Joseph ideal. Thus H is a so-called minimal representation. The papers [60J and [64J have had enormous influence on the study of minimal representations.

The paper [64J also contains the marvelous fact that in dimension 6, and only in dimension 6, is the vanishing of the scalar curvature a con­formally invariant condition. It also discusses a theorem of Biedrzycki (a student of Kostant) on constructing solutions of the Einstein equations on compactified Minkowski space by embedding it into S3 x S3.

In the book [BI], Kostant develops the idea that both the gravitational field and the electromagnetic field on compactified Minkowski space M can be obtained from an embedding M "--> X, where X is equipped with a conformal structure of signature (3,3). The gravitational field is obtained from the induced Lorentzian metric on M, and the electromagnetic field on M is obtained from the connection on the normal bundle induced by the Levi-Civita connection on X. There are three different models of confor­mally flat completions of 1R3 ,3, which are related by triality. In each model there is a natural embedding of M. This sort of Kaluza-Klein picture of gravitation and electromagnetism is called the "Kostant universe" in the book Variations on a theme of Kepler by V. Guillemin and S. Sternberg.

The paper [56J of Kostant and Sternberg is a mathematical treatment of the theory BRS quantization (named after 13ecchi, Rouet and Stora), which is of great importance in mathematical physics (in particular, in gauge theory and in string theory). The first result of [56J is a homological construction of the space of functions on the Marsden-Weinstein reduction B of a symplectic manifold M with respect to a Hamiltonian action of a lie group G. The space F(B) of functions on B is the degree 0 cohomology group of the complex Ag* 0Ag0F(M), equipped with the BRS differential

Preface xxix

D. Then Kostant and Sternberg make the crucial observation that since a = gEeg* is an orthogonal vector space, one can put on Ag*®l\g a structure of associative algebra, the Clifford algebra of o. Then I\g* ® I\g ® F( M) becomes a Poisson superalgebra. A wonderful result of [56] is that the differential D is given by Poisson bracket with a canonical element e of this Poisson algebra. The total degree in the complex is also given by a Poisson bracket. Then [56] studies the conditions under which {8,8} is a cOinstant (or equivalently, D has square zero), which are quite delicate in the infinite-dimensional case, and which is a case of importance in physics. One new feature of the infinite-dimensional case is that there are many different types of Spin representations. The article of Ding and Frenkel in this volume contains the construction of analogs of the Spin representation for quantum groups.

Finally [56] discusses so-called BRS quantization, which involves ten­sOiring with the spin representation of the Clifford algebra of o. BRS coho­mology and its applications in conformal field theory are discussed in the paper of Lian and Zuckerman in this volume. The related theory of vertex operator algebras is the topic of the article by Huang and LepowskYi this theory was developed by 1. Frenkel, Lepowsky and Meurman in connection with the monster group .

BRS quantization is further developed in C. Duval, J. Elhadad, G. M. Tuynman, "Comm. Math. Phys. 1126 (1990).

The paper [59], with Guillemin and Sternberg, recasts the solution of the Plateau problem by J. Douglas in terms of the action of Diff+ (S1) on the space of vector-valued functions on S1. There are some interesting formulas in [59] which deserve further study.

The paper [61] with Sternberg gives a geometric interpretation of the classical Schwartzian derivative of a function in terms of the structure of the torus S1 x S1 under the diagonal action of Diff+ (S1 ). There are two orbits: an open orbit C, diffeomorphic to an open cylinder, and a closed orbit D (the diagonal of 8 1 x 8 1), diffeomorphic to 8 1• There is a natural (up to constant) PSL(2, lR)-invariant measure J.l on C, since C identifies with a coadjoint orbit of P8L(2,lR). Given 7 E Diff+(81), it admits a Radon-Nikodym derivative IT> such that J.l7 = IT· J.l, and IT extends to smooth function on 8 1 x 8 1 which vanishes on D and has vanishing normal derivative along D. Then the Schwartzian derivative of 7 is the second normal derivative of IT along D. This easily gives the cocycle property of the Schwartzian derivative. This work has intriguing connections with string theory and conformal field theory.

Kostant proved in [62] an equality expressing, for an arbitrary semisim­pIe Lie group G, the trace of the product of an intertwining operator and a

xxx Preface

certain bounded operator as a sum over the K-types. The trace is equal to the inverse of the well-known c-function. The resulting equality specializes, in the case of SO(n, 1), to an identity of Kummer expressing certain values of hypergeometric functions in terms of the r function. [62] contains a dis­cussion of the relation of the Kunze-Stein operators with the p"Y matrices of [35], as well as a theorem on the traceability of intertwining operators.

Continuing his work on the Amitsur-Levitski theorem in [49], Kostant related the whole subject of that identity to the theory of transgression in the Leray spectral sequence for the fibration G -+ EG -+ BG. There are a number of further unpublished results of Kostant in this direction having to do with a Clifford algebra analog of the Hopf-Koszul theorem which says that the cohomology ring of a Lie algebra is an exterior algebra.

An important result, obtained by Kostant around 1980, gives quadratic equations for the closure of the G-orbit of a highest weight vector v in a finite-dimensional irreducible representation VA of a simple Lie group G. Specifically, a non-zero vector u belongs to the orbit G . v if and only if the vector u®u of VA ® VA belongs to the irreducible component V2'A of VA ® VA. Kostant then writes down explicit quadratic equations as follows: if C is the Casimir operator, and y the scalar by which C operates on VA' then u belongs to G· v if and only if C· (u ® u) = y. (u ® u). This a far-reaching generalization of the Plucker equations which characterize decomposable vectors inside I\PCn . In terms of projective geometry, Kostant gives explicit quadratic equations for any flag manifold G / P inside a projective space IP'(VA).

This theorem of Kostant is recalled and used in the thesis of D. Garfin­kle, MIT, 1982. It was generalized to affine Kac-Moody groups by Date, Jimbo, Kashiwara and Miwa; the quadratic equations become the Hirota bilinear wave equations for the T-function, which characterize the solu­tions to the KdV type hierarchies. These equations play a role in the Kac-Peterson theory of Kac-Moody groups.

The results of Bernstein-Gelfand-Gelfand and of Demazure on the co­homology and K-theory of flag manifolds were generalized to the case of an arbitrary Kac-Moody group G in the two papers [55] [58] in collaboration with S. Kumar. The Kac-Moody groups considered in [55] [58] are those constructed by Kac and Peterson. A main object introduced in [55] is the nil Heeke ring R, which is a subring of the smash-product of the group ring of W with the field of rational functions on a Cartan subalgebra ~. [58] contains a structure theorem for the ring R. The cohomology of the flag manifold G / B is described combinatorially in terms of the ring Rand its "dual" A, which is defined explicitly in terms of the Coxeter group W acting on ~. In the paper [58] Kostant and Kumar introduce another ring

Preface xxxi

Y inside the smash-product of Z[W] with the field of rational functions on the maximal torus T, and its dual W. They prove that W identifies with the equivariant K-theory Kr(GIB). They can then describe explic­itly various operations on equivariant K-theory analogous to those studied by Demazure in the finite-dimensional case, and describe the equivariant K-theory of the Bott-Samelson-Demazure varieties. Kostant and Kumar prove that the K-theory K(GIB) itself is a quotient of Kr(GIB) which corresponds to a specific specialization of A.

The work of Kostant and Kumar has had a vast influence on the study of flag manifolds and Schubert varieties, including the finite-dimensional case. Kumar used [58] in his study of characters of affine Kac-Moody groups; similar results were obtained independently by O. Mathieu. A. Arabia has studied the T -equivariant cohomology of G I B and of Schubert varieties. Kumar has used the nil Hecke ring in developing criteria for smoothness and rational smoothness of Schubert varieties. The paper of Kumar in this volume represents his further work, and contains references to other important works by Deodhar, Carrell, D. Peterson, Polo, M. Dyer, and others.

It should also be noted that equivariant K-theory and equivariant co­homology have become a major instrument of geometric quantization and of representation theory. We will mention the work of Borho, J-L. Brylinski and MacPherson on primitive ideals, the work of Kazhdan-Lusztig and of Ginzburg on representations of affine Hecke algebra and of p-adic groups, the work of Berline-Vergne, of Rossmann and of Duflo-Heckman-Vergne on the Kirillov conjecture and the work of Duflo-Vergne (and also Kumar) on equivariant cohomology, invariant distributions and characters of represen­tations, for which we will refer the reader to the article of Duflo and Vergne in this volume.

In the early 80s Kostant found a beautiful representation-theoretic framework for the McKay correspondence between finite subgroups of 8L(2, q (up to conjugacy) and simply-laced complex simple Lie algebras. Recall that for a finite subgroup r of 8L(2, C), one can parametrize the irreducible representations 7I"i of r by the vertices or simple roots ai of the extended Dynkin diagram of a simple Lie algebra g, in such a way that if Cij is the affine Cartan matrix, the tensor product by the tautological two-dimensional representation 71" of r is given by the formula

71" ® 7I"i = L (2 - C ij )· 7I"j.

j

On the other hand the Dynkin diagram in question can be recovered from r by means of algebraic geometry, as was observed by M. Artin in

xxxii Preface

the 70s. The quotient algebraic variety Y = ((:2/r is a normal algebraic surface with an isolated singular point 0, and there is a minimal resolution of singularities 71" : Y -+ Y. The "exceptional fibre" 71"-1(0) is a union of rational curves D i , which correspond to the vertices of the Dynkin diagram, in such a way that the Cartan matrix is Cij = -Di . D j , where Di n Dj denoted an intersection number in Y.

Kostant's work in [53] goes a long way toward unifying these two points of view. To explain some of the results, choose the parametrization of the 7I"i in such a way that 71"0 is the trivial representation. It is known from algebraic geometry that the generating function of the graded algebra of r-invariants in S«((:2) is equal to the quotient ( 1 ;/h b)' where h is the

I-ta I-t Coxeter number and a, b are integers such that a+ b = h+ 2 and a· b = 2 ·Irl. Kostant studies the generating function Pi(t) for the multiplicities of any 7I"i in the symmetric powers sn«((:2). Kostant proves that

where Zi(t) is a polynomial which is determined explicitly as follows: Let CT

be a Coxeter element in Wand let D. be the set of roots of g. Then there are l orbits of CT in D., each of which has h elements. For a distinguished choice of CT, the D.i are naturally parametrized by the simple roots (ti (for i > 0). On the other hand there is the height function h on D.. Let <P c D. be the set of roots which have a positive inner product with the highest root and let <Pi = <P n (D. - i). Then if i > 0 is not a branch point one has

Zi(t) = L tn(t/». t/>E4>;

For the branch point of the D and E diagrams, the formula is even simpler.

Kostant in fact was always fascinated by the exceptional Lie algebras and groups. This led him to study some amazing embeddings of finite simple groups into exceptional groups, which are the topic of the paper [63]. To give an example, Kostant shows that the 248-dimensional Lie algebra Es decomposes as the direct sum of 31 Cartan subalgebras, which he considers as the 31 points of the projective plane over lFs . The symmetry group PSL(3, 5) of this finite geometry then embeds into the Lie group E s, and this allows Kostant to derive beautiful formulas for the bracket in the Lie algebra in terms of Gauss sums. In a similar fashion, Kostant shows that the 13 points of the projective plane over lFa embed PSL(3,3) into

Preface xxxiii

F4 , and the 7 points of the projective plane over 1F2 embed PSL(3, 2) into G2 •

The paper [63] also contains the conjecture that if G is a simple compact Lie group, with Coxeter number, for which 2h+ 1 is prime, then the group SL(2,2h + 1) embeds into G; this conjecture is verified in [63] in many cases. The last open case of the conjecture, that of E8 , was solved recently by Griess and Cohen.

90s

Kostant and Sahi found in the papers [66] and [70] a remarkable general­ization of the classical Capelli identity in the context of symmetric spaces. They give a new interpretation of that identity based on Jordan algebras. For a symmetric space G / K with corresponding Cartan decomposition g := t E9 \" it is known that the algebra of G-invariant differential oper­ators on G / K is isomorphic to the ring S( a) w, where a C \' is a maximal abelian subspace. Now let V be an irreducible finite-dimensional repre­sentation of G which admits a K-invariant vector v. Then G/K embeds into V; if G / K is of co dimension 1 in V, then its dilates span an open set in V, hence any operator in R defines a differential operator on V of degree 0. This co dimension 1 condition is not frequently satisfied, but it certainly holds true (by the Tits-Koecher theory) whenever V has a Jor­dan algebra structure, v is the identity of the Jordan algebra, and G is the "norm-preserving group". Assuming this to be the case, let p E S(V) be the Jordan norm, and let op be the corresponding constant coefficient differential operator. Then p . op is a G-invariant differential operator on V of degree 0. Kostant and Sahi identify this operator with that induced by an explicit element {j of S(a)w. They are then able to compute the eigenvalues of p . op acting on the various K-types in S(V). The original Capelli identity is recovered from the Jordan algebra of n x n-matrices. The identities of [66] [70] have found many applications in the theory of minimal representations, and are now being used by I. Gelfand and others in their work on free algebras. Another approach to proving the Capelli identity and similar identities is given by Sahi in his contribution to this volume.

In [67], R. Brylinski and Kostant determine completely the structure of the set Symp(M) of all G-invariant symplectic structures on any homoge­neous space M of G, for G a complex semisimple Lie group. By a theorem of Kostant in [28], any G-homogeneous symplectic manifold is a covering of a coadjoint orbit 0 = G· x. The question reduces to the case where 0 is nilpotent. Some of the main results are:

xxxiv Preface

- Symp(M) is naturally isomorphic to Ox = On gX; - Ox is Zariski dense in gX and its complement is a finite union of

hyperplanes; - Ox has a group structure, and is isomorphic (up to a finite cover)

to the centralizer of G in Diff( M). - for X principal nilpotent, if N(GX) is the normalizer of GX, then

dimN(GX) = 2· dimGx.

In [69] (announced in [68]), R. Brylinski and Kostant discover new phenomena involving nilpotent orbits by combining techniques of algebraic and symplectic geometry. The basic object is a G-homogeneous covering M of a complex nilpotent orbit a so that M is both a complex algebraic variety and a symplectic manifold with a Hamiltonian G-action. Here G is a complex simply-connected semisimple algebraic group with Lie algebra g. The ring R = E9pcoRIP] of regular functions on M is then a graded Poisson algebra where (with respect to the natural G-invariant grading) the Poisson bracket has degree -2 and the Hamiltonian action defines an inclusion g C R[2]. One of the main discoveries is that R "sees" a maximal (algebraic) symmetry group G' which contains the action of G. More precisely, they prove that the space of functions g' = R[2] is a (in fact, the unique maximal) finite-dimensional semisimple Lie subalgebra of R containing g. Let G' be the simply-connected Lie group corresponding to g'. They show that there exists a unique nilpotent orbit A' of G' and a G'-homogeneous cover M' of A' such that M' contains M as an open dense set (with boundary of codimension at least 2) and the infinitesimal Hamiltonian action of g' on M integrates to an action of G' on M'. The two orbits M and M' then have the same ring of regular functions and share many geometric properties; Brylinski and Kostant call (M, M') a shared orbit pair if g i= g'.

To construct the G'-action, they consider the variety X = Spec(R). They prove X is a normal affine variety which is equal to the normalization of a in the function field of M. The action of G on X has finitely many orbits and extends to an action of G'. In fact M and M' are, respectively, the unique open dense orbits of G and G' on X. There is a unique G-fixed point 0 in X, and there is G-equivariant embedding X <-> T; X which is dimension-wise a minimal embedding of X into an affine space. Except in the metaplectic case, R[l] = 0 and X is singular; then X is the closure of a coadjoint orbit of G' if and only g' = T; X.

There are a number of interesting cases where g' is strictly larger than g; these are completely classified in [69]. A beautiful example occurs when g =

sZ(3, q and M is the 3-fold simply-connected cover of the 6-dimensional

Preface xxxv

principal nilpotent orbit. Then g' is the 14-dimensional exceptional simple Lie algebra of type G2 and M' is the minimal (non-zero) nilpotent orbit in g'. Another striking example provided much of the motivation for [68) [69): Levasseur and Smith, proving a conjecture of Vogan, discovered that the normalization of the closure of 8-dimensional nilpotent orbit M = 08 of G2 is equal to the closure of the minimal nilpotent orbit Oroin of 80(7, C); thus (08 ,Oroin) is a shared orbit pair. In this case 0 8 is the orbit Os of a short vector; Brylinski and Kostant find that for the other doubly-laced simple Lie algebras the simply-connected 2-fold cover of Os lies in a shared orbit pair. They also prove several general results, e.g., g' is a simple Lie algebra if 9 is simple. The full classification of all such pairs (g, g') is related to the well-known occurrence of 2 root lengths for a simple Lie algebra.

The study of "hidden symmetries" in [69) has interesting connections with work of Souriau and others on the Kepler problem.

The papers [72) [74) [75) of R. Brylinski and Kostant break new ground by showing that the machinery of geometric quantization, suitably gener­alized, works for the so-called minimal nilpotent orbit of a semisimple Lie group and produces a unitary minimal representation of GR (in particular the representation is irreducible and its annihilator is the Joseph ideal). An example is the Fock space model of the metaplectic representation. Vo­gan showed that there are strong restrictions on the K-types of minimal representations. The models of the minimal representations constructed by Brylinski and Kostant are in the spirit of the Fock model, as the ac­tion of K is very natural and explicit in their models. Other models for most of the representations of [74) [75) (of a rather different nature) had already been constructed by Vogan, Kazhdan-Savin, by Gross-Wallach (cf. their article in this volume), by Kostant in [60) [64), and by Binegar-Zierau (precise references can be found in [74)). In particular, Gross and Wallach obtain beautiful results on discrete restriction of the minimal representa­tions to subgroups. Many mathematicians and physicists have constructed very interesting examples of minimal, and more generally singular, repre­sentations.

The papers [72] [74) [75) are based on far-reaching ideas from algebraic geometry and symplectic geometry, and give beautiful and explicit models for the Harish-Chandra modules where 9 acts by explicit pseudo-differential operators. They give a uniform construction of minimal representations in all cases where they exist, including the non-spherical minimal represen­tations. In the spirit of geometric quantization, the geometry is based on the lagrangian subvariety Y of the minimal orbit 0 n p* (there is no loss of generality in assuming Y not empty, since no minimal representation can exist if Y is empty). Here 9 = t+p is a complexified Cartan decomposition

xxxvi Preface

corresponding to the real simple simply-connected Lie group Gilt such that 'Y is simple and the pair (g, t) is non-Hermitian. The Harish-Chandra mod­ule is to be H = r(Y, N~), where N~ is a half-form bundle over Y. The Lie algebra t acts naturally on H, and the difficulty lies in constructing operators associated to elements of p. The difficulty is that 0 admits no G-invariant polarization. However [74] gives the amazing result that the cotangent bundle T*Y and a ramified double cover of 0 share an open set M; this defines an infinitesimal action of g on M c T*Y, which is Hamil­tonian. Brylinski and Kostant go further in [74] to show that the function on M associated to any v E p decomposes as fv - gv, where the fv generate the Poisson commutative subalgebra R(Y) of R(M), and the gv generate another (transverse) Poisson commutative subalgebra.

The construction of the operators associated to elements of p, given in detail (in a uniform way) [74], converts the functions fv - gv on Minto pseudo-differential differential operators on H. While fv obviously defines a multiplication operator, gv defines a pseudo-differential differential oper­ator which is equal to (E'(E' + 1))-1 Dv where Dv is an order 4 differential operator while E' is the Lie derivative with respect to the Euler vector field on Y (they prove that E' has positive spectrum on H). The proof that these operators form a Lie algebra isomorphic to g combines their results on the geometry of Y and T*Y with a skillful application of the work of Kostant and Sahi on the generalized Capelli identity; a crucial point is that a certain subspace of t is always a Jordan algebra of rank S 4 and further­more there is a monomial P in the Jordan norms of degree exactly 4. The paper [75] (in this volume) contains the beautiful construction of the new differential operators Dv. If e E p is the highest weight vector then De is the quotient of an operator corresponding to P by the linear function on Y defined bye.

The explicit and uniform construction of the· minimal representations by means of geometric quantization allows Brylinski and Kostant to give a beautiful description of the invariant hermitian form. In [72] and [74] this hermitian form is written down in terms of hypergeometric functions. The methods of [72] [74] [75] open the road to a study of the minimal representations from the point of view of harmonic analysis; already in [74] there is a formula expressing the spherical function restricted to a root TDS.

The methods of [72] [74] [75], which are based on geometry, have the po­tential to carryover to more general nilpotent orbits; Brylinski and Kostant have embarked on their program to do this.

Kostant has also made great progress in the last few years in a vast new program of relating the group PSL(2, 11) to the polyhedron called

Preface xxxvii

buckminseterfullerene (or simply buckyball). This is a truncated icosahe­dron which has 60 edges and 90 vertices. Of these edges, 60 edges (referred to as pentagonal) belong to the boundary of the 12 pentagons surrounding the vertices of the icosahedron, and the remaining 30, referred to as hexag­onal, bound only hexagons. The graph r made of these vertices and edges materializes in the coatings of many viruses and in the structure of the carbon molecule G60 • The polyhedron itself was used a number of years ago by the architect Buckminster Fuller. The fact that the 60-element icosahedral group A acts as a symmetry group leads one to believe that noncommutative harmonic analysis can explain some of the chemistry of these molecules. This was already the case in work of F. Chung and S. Sternberg, who deduced the vibrational spectrum from group theory. In the paper [73] by Chung, Kostant and Sternberg, appearing in this vol­ume, the geometry of the buckyball is studied further in connection with the group PSL(2, 11); it is shown in [73] that the 12 vertices of the icosahe­dron can be identified naturally with the projective line Pl1, and the action of PSL(2, 11) extends naturally to an action on the 90 vertices of r (the reason being that the edges of the icosahedron have a beautiful interpreta­tion in terms of the cross-ratio in P l1 ). The action of PSL(2, 11) preserves the pentagonal edges but not the hexagonal edges. The paper [73] also de­scribes the embedding A ~ PSL(2, 11) and the corresponding branching rules, as well as their relation to the electronic spectrum. Then the paper discusses the Huckel model modified to include the magnetic field, and the possible rule of the double cover of A in the physics of the Fullerene.

The very recent paper [76] goes much further by describing the en­tire structure of the buckyball graph r in terms of PSL(2, 11). Kostant identifies the set of vertices of r with a conjugacy class M in PSL(2,11) consisting of elements of order 11; then the two pentagonal edges having x E M as a vertex are {x,x3} and {x,x-3}, and the unique hexagonal edge having x as a vertex is {x,Px}. Here Px = x-I. (lx' X· (lx, where (lx is the unique element of order 2 of A c PSL(2,11) such that the above expression is again an element of order 2 of A. Thus the entire graph r is constructed from group theory. This should pave the way to using group representations to further study the physics and chemistry of the molecule G450.

The recent paper [71] gives a beautiful generalization of a positivity theorem of J. Stembridge which shows that if an n x n- matrix A is totally positive (Le., all its minors are 2: 0), then for any partition A of n the immanant ImmA(A) is 2: O. First it is shown in [71] that the immanant can be written as

ImmA(A) = Tr PA7fA(A)PA,

xxxviii Preface

where 1I"A : GL(n) -+ Aut(VA) is the representation of highest weight A, and PA is the projection of VA to the zero weight space. Then the generalization of the Stembridge theorem is the inequality

where 11" : GL(n) -+ Aut(V) is any finite-dimensional representation of GL(n), and 11" : V -+ VH is the H-equivariant projection onto the space V H of H-invariant vectors, for H a maximal torus.

The proof of this theorem uses a result of A. Whitney on the structure of the semigroup of totally positive matrices, together with a deep positivity result of Lusztig concerning his canonical basis (cf. the paper of Lusztig in this volume).

When we last spoke with Bert he was busy, as always, working on many projects.

We have all benefited immensely, not only from reading many of Kostant's papers, but also from so many exciting lectures and inspiring conversations. We wish him the best for the next century.

The editors warmly thank everyone at Birkhauser and in particular Elizabeth Hyman. We are very grateful to Ann Kostant for all her guidance, support and patience in preparing this volume.

Published Works of Bertram Kostant

1. Holonomy and the Lie Algebra of Infinitesimal Motions of a Rieman­nian Manifold, Trans. Amer. Math Soc. 80 (1955), 528-542.

2. On the Conjugacy of Real Carlan Subalgebras, I, II, Proc. Acad. of Sci. 41: 11, Nov. 1955.

3. On Invariant Skew Tensors, Proc. Nat. Acad. of Sci. 42: 3, March 1956, 148-151.

4. On Differential Geometry and Homogeneous Spaces, I, II, Proc. Nat. Acad. Sci. 42 (1956), 258-261, 354-357.

5. On Holonomy and Homogeneous Spaces, Nagoya Math. J. 12(1957), 31-54.

6. A Theorem of Probenius, A Theorem of Amitsur-Levitski and Coho­mology Theory, J. of Math. and Mech. 7: 2 (1958),237-264.

7. A Characterization of the Classical Groups, Duke Math. J. 25: 1(1958), 107-124.

8. A Formula for the Multiplicity of a Weight, Trans. Amer. Math. Soc., 93: 1 (1959), 53-73.

9. The Principal Three Dimensional Sub-Group and the Betti Numbers of a Complex Simple Lie Group, Am. J. Math., Oct. 1959,973-1032.

10. A Characterization of Invariant Affine Connections, Nagoya Math. Jour. 16 (1960), 35-50.

11. Lie Algebra Cohomology and the Generalized Borel- Weil Theorem, Ann. of Math. 74: 2 (1961),329-387.

12. (with G. Hochschild and A. Rosenberg), Differential Forms on Regular Affine Algebras, Trans. Amer. Math. Soc. 102: 3 (1962),383-408.

13. (with G. Hochschild), Differential Forms and Lie Algebra Cohomology for Algebraic Linear Groups, III, Jour. of Math. 6(1962), 264-281.

14. Lie Group Representations on Polynomial Rings, Bull. Amer. Math. Soc. 69(1963), 518-526.

15. Lie Group Representations on Polynomial Rings, Am. J. Math. 85 (1963), 327-404.

16. Lie Algebra Cohomology and Generalized Schuberl Cells, Ann. of Math. 77: 1 (1963), 72-144.

17. (with A. Novikoff), A Homomorphism in Exterior Algebra, Canad. J. Math. 16 (1964), 166-168.

18. Eigenvalues of a Laplacian and Commutative Lie Subalgebras, Topol­ogy, 13 (1965), 147-159.

xl Published Works of Bertram Kostant

19. Orbits, Symplectic Structures and Representation Theory, Proc. of U.S.-Japan Seminar in Differential Geometry, Kyoto, Japan, 1965.

20. Groups Over Z, Proc. Symposia in Pure Math., 9 (1966), 90-98.

21. (with L. Auslander), Quantization and Representations of Solvable Lie Groups, Bull. Amer. Math. Soc. 73: 5 (1967),692-695.

22. Irreducibility of Principal Series and Existence and Irreducibility of the Complementary Series, Bowdoin Conf., Aug. 1968, 1-41.

23. (with Stephen Rallis), On Orbits Associated with Symmetric Spaces, Bull. Amer. Math. Soc. 75 (1969),879-883.

24. (with Stephen Rallis), Representations Associated with Symmetric Spaces, Bull. Amer. Math. Soc. 75 (1969),884-888.

25. On the Existence and Irreducibility of Certain Series of Representa­tions, pub. as Invited Address in Bull. Amer. Math. Soc., May 1969, 627-642.

26. On Certain Unitary Representations which arise from a Quantization Theory, Lecture Notes in Phys., Vol. 6, Battelle Seattle Rencontres, Springer-Verlag, 1970, 237-254.

27. Orbits and Quantization Theory, Proc. Int. Congress of Mathemati­cians, Nice, 1970, 395-400.

28. Quantization and Unitary Representations, Lecture Notes in Math. 170 (1970), Springer-Verlag, 87-207.

29. (with Stephen Rallis), Orbits and Representations Associated with Symmetric Spaces, Amer. Jour. of Math. 93 (1971), 753-809.

30. (with L. Auslander), Polarization and Unitary Representations of Solv­able Lie Groups, Inventiones Math., 1971, 255-354.

31. Line Bundles and the Prequantized Schrodinger Equation, ColI. Group Theoretical Methods in Physics, Centre de Physique Theorique, Mar­seille, June 1972, 81-85.

32. On Convexity, the Weyl Group and the Iwasawa Decomposition, Ann. Sci. Ecole Norm. Sup. 6: 4 (1973), 413- 455.

33. Symplectic Spinors, in: Symp. Math., Vol. XIV, Instituto Naz. di Alt. Mat. Roma., Academic Press, London-New York 1974, 139-152.

34. Verma Modules and the Existence of Quasi-Invariant Differential Op­erators, Lecture Notes in Math. 466(1974), Springer-Verlag, 101-129.

35. On the Existence and Irreducibility of Certain Series of Represen­tations, in: Lie Groups and Their Representations, edited by I. M. Gelfand, Summer School Conf. Budapest, 1971, Halsted Press, Wiley Press, 1975, 231-331.

36. On The Tensor Product of a Finite and an Infinite Dimensional Rep­resentation, J. Funct. Anal. 20: 4 (1975), 257-285.

Published Works of Berlmm Kostant xli

37. On the Definition of Quantization, Geometrie Symplectique et Physique Mathematique, ColI. CNRS, No. 237, Paris, 1975, 187-210.

38. (with D. Sullivan), The Euler Characteristic of an Affine Space Form is Zero, Bull. Amer. Math. Soc. 81: 5 (1975).

39. On MacDonalds .,.,-Function Formula, the Laplacian and Generalized Exponents, Adv. in Math., 20: 2 (1976), New York, 179-212.

40. (with J. Tirao) On the Structure of Certain Sublalgebras of a Universal Enveloping Algebra, Trans. Amer. Math. Soc. 218(1976), 133-154.

41. Graded Manifolds, Graded Lie Theory, and Prequantization, Lecture Notes in Math. 570 (1977), Springer-Verlag, 177-306.

42. Quantization and Representation Theory, in: Representation Theory and Lie Groups, Proc. SRC/LMS Res. Sympos., Oxford, 1977, pp 287-316; London Math Soc. Lecture Note Ser 34, Cambridge Univ Press, Cambridge-New York, 1979.

43. Harmonic Analysis on Graded (or Super) Lie Groups, Group Theoreti­cal Methods in Physics, Sixth International Colloquium, Lecture Notes in Physics, Springer, 1979, 47-50.

44. On Whittaker Vectors and Representation Theory, Inventiones Math. 48 (1978), 101-184.

45. (with D. Kazhdan and S. Sternberg), Hamiltonian Group Actions and Dynamical Systems of Calogero Type, Communications Pure and Ap­plied Math., 31:4 (1978), 483-507.

46. The Solution to a Generalized Toda Lattice and Representation Theory, Adv. in Math., 34 (1979), 195-338.

47. Poisson Commutativity and Generalized Periodic Toda Lattice, Differ­ential Geometric Methods in Math. Physics, Lecture Notes in Mathe­matics 905 (1980), Springer-Verlag, 12-28.

48. (with S. Sternberg), Symplectic Projective Orbits, New Directions in Applied Mathematics, Springer-Verlag, (Cleveland, Ohio, 1980), Springer-Verlag, 1982, 81-84,

49. A Lie Algebra Generalization of the Amitsur-Levitski Theorem, Adv. in Math. 40 : 2 (1981), 155-175

:50. Coadjoint Orbits and a New Symbol Calculus for Line Bundles, Conf. on Diff. and Geometric Methods in Theoretical Physics, Trieste 1980, World Scientific Publishing, 1981, 66-68.

:51. The Coxeter Element and the Structure of the Exceptional Lie Groups, ColI. Lectures of the AMS, 1983, Notes available from the AMS.

52. On Finite subgroups of SU(2), simple Lie Algebras, and the McKay Correspondence, Proc. Nat. Acad. Sci. U.S.A., 81:16 (1984), Phys. Sci., 5275-5277.

xlii Published Works of Bertmm Kostant

53. The McKay Correspondence, the Coxeter Element and Representation Theory, Societe Math. de France, Asterisque, hors series, 1985, 209-255.

54. (with S. Kumar), The Nil Hecke Ring and Cohomology of GIP for a Kac-Moody Group G*, Proc. Nat. Acad. Sci. USA. 83:6 (1986), 1543-1545.

55. (with S. Kumar), The Nil H ecke Ring and Cohomology of G / P for a Kac-Moody Group G*, Adv. in Math. 62 (1986), 187-237.

56. (with S. Sternberg), Symplectic Reduction, BRS Cohomology and In­finite Dimensional Clifford Algebras, Annals of Physics 176 (1987), 49-113.

57. (with S. Kumar), T-Equivariant K -Theory of Generalized Flag Vari­eties, Proc. Nat. Acad. Sci. U.S.A. 84:13 (1987),4351-4354.

58. (with S. Kumar), T-Equivariant K-Theory of Generalized Flag Vari­eties, Journal of Differential Geometry, 32 (1990), 549-603.

59. (with V. Guillemin and S. Sternberg), Jesse Douglas' Solution to the Plateau Problem, Proc. Nat. Acad. Sciences, May 1988, 3277-3278.

60. The Principle of Triality and a Distinguished Unitary Representation of SOC 4, 4), Differential Geometrical Methods in Theoretical Physics, edited by K. Bleuler and M. Werner, Kluwer Academic Publishers, 1988, 65-108.

61. (with S. Sternberg), The Schwartzian Derivative and the conformal geometry of the Lorentz hyperboloid, Quantum Theories and Geometry, edited by M. Cahen and M. Flato, Kluwer Academic Publishers, 1988, 113-125.

62. A Formula of Gauss-Kummer and the Trace of Certain Intertwining Operators, The Orbit Method in Representation Theory, edited by Duflo, Pedersen and Vergne, Birkhauser, 1990, 99-134.

63. A Tale of Two Conjugacy Classes, in preparation.

64. The Vanishing of Scalar Curvature and the Minimal Unitary Represen­tation of SO(4,4), Operator Algebras, Unitary Representations, En­veloping Algebras, and Invariant Theory, edited by Connes et aI, PM 92, Birkhauser-Boston,1990, 85-124.

65. (with Shrawan Kumar), A Geometric Realization of Minimal K-Types of Hansh-Chandra Modules, Kazhdan-Lusztig Theory and related Top­ics, Contemporary Math., edited by V.V. Deodhar, Vol. 139, 1992, Providence.

66. (with Siddhartha Sahi), The Capelli Identity, Tube Domains and a Generalized Laplace Transform, Adv. in Math. 87 (1990), 71-92.

Published Works of Bertmm Kostant xliii

67. (with Ranee Brylinski), The Variety of all Invariant Symplectic Struc­tures on a Homogeneous Space, Symplectic Geometry and Mathemati­cal Physics, PM 99, edited by Donato et al. Birkhauser-Boston, 1991, 80-113.

68. (with Ranee Brylinski), Nilpotent Orbits, Normality and Hamiltonian Group Actions, Bull. AMS. 26 (1992), 269-275.

69. (with Ranee Brylinski), Nilpotent Orbits, Normality and Hamiltonian Group Actions, JAMS 7 (1994), 269-298.

70. (with Siddhartha Sahi), Jordan Algebras and Capelli Identities, Inven­tiones Math., 112 (1993), 657-664.

71. Immanant Inequalities and O-weight Spaces, Tto appear, JAMS.

72. (with Ranee Brylinski), Minimal representations of E6 , E7 and Es and the Generalized Capelli Identity, Proc. Natl. Acad. Sci. USA 91 (1994), 2469-2472.

73. (with F. Chung and S. Sternberg), Groups and the Buckyball, to ap­pear in: Lie Theory and Geometry, edited by J.-L. Brylinski, R. Brylin­ski, V. Guillemin, V. Kac, PM 123, Birkhauser, 1994

74. (with Ranee Brylinski), Minimal representations, geometric quantiza­tion and unitarity, Proc. Natl. Acad. Sci. USA 91 (1994), 6026-6029.

75. (with Ranee Brylinski), Differential operators on conical lagrangian manifolds, to appear in: Lie Theory and Geometry, edited by J.-L. Brylinski, R. Brylinski, V. Guillemin, V. Kac, PM 123, Birkhauser, 1994.

76. Structure of the truncated icosahedron (e.g. fullerene or C60, viral coat­ings) and a 60-element conjugacy class in PSL(2,11), 1994, Selecta Math., to appear

Books

Bl A Course in the Mathematics of General Relativity, ARK Publications, 1988

Normality of Some Nilpotent Varieties and Cohomology of Line Bundles on the

Cotangent Bundle of the Flag Variety

Bram Broer

Dedicated to Bert Kostant on his 65th birthday

Introduction

Let G be a connected, complex reductive group and B, a Borel subgroup. The homogeneous space G / B is called the (full) flag variety of G; for G Ln this is the ordinary variety of flags of subspaces in en. The properties of this variety have many implications in the study of reductive groups and their representations. For example, all the irreducible finite dimensional G-modules can be obtained as global sections of line bundles on the flag variety. More generally, any of its (G-linearized) line bundles has at most one non-vanishing sheaf-cohomology space, having a natural structure of a simple G-module if G is simply connected. This is part of the content of the Borel-Weil-Bott theorem and can be seen as a geometric interpretation of E. Cartan's theory of highest weights.

The properties of the cotangent bundle T* G / B on G / B also have many ramifications. First of all, it has a canonical symplectic structure and an associated moment map J.L: T*G/ B -+ g*, where g* is the dual of the Lie algebra g of G. After identifying g* with g using the Killing form the image can be identified with the cone N of nilpotent elements in g. This is in itself a very interesting singular variety with finitely many G-orbits. The closures of these orbits are called nilpotent varieties. The full nilpotent variety is normal by B. Kostant, and the moment map is T.A. Springer's resolution of singularities. By a result of W. Hesselink the higher sheaf cohomology of the structure sheaf of T*G/B vanishes, which implies that .N has only rational singularities. The top homology groups of the fibres of J.L carry the Springer representations of the Weyl group. Any sheaf V of twisted differential operators on G / B has a filtration, with associated graded sheaf isomorphic to the structure sheaf of T* G / B. Hesselink's result implies that V has vanishing higher cohomology. Starting from A. Beilinson and J . Bernstein's paper [2] much has become known about the relations between V-modules and subvarieties of T*G / B.

2 Bmm Broer

For SLn all nilpotent varieties are normal; this is not the case for general simple algebraic groups, not even for the classical groups as SOn and SPn (results obtained by C. Procesi and H. Kraft); for the exceptional groups many nilpotent varieties are known to be not normal. For a recent overview see the appendix in [15]. Most results on the normality prob­lem use the Dynkin-Kostant-classification of nilpotents, with as exceptions Kostant's result that the full nilpotent cone is normal, our earlier result that the subregular cone is normal and results by Hesselink, Kempf, Popov and Vinberg stating that small nilpotent varieties are normal. In this article we give a uniform proof of the normality of the closure of the Richardson orbit associated to a parabolic subgroup with Levi factor of semisimple type At x ... X Al corresponding to short simple roots. The regular and subregular nilpotent varieties can be obtained using this construction. For the proof we need that the i-th cohomology groups vanish for i big enough for certain line bundles on T* G / B .

In [6] R.K. Brylinski started the study of certain ideals in C[N] and con­jectured connections with global sections of some line bundles on T* G / B. Her conjectures were shown by us in [5]; a characterization of the line bun­dles was given with vanishing higher cohomology and for these line bundles we gave a description of the module of global sections in terms of Brylinski's ideals. We shall extend these results in various ways.

This article starts by showing that dominant line bundles on T* G / P, where P is parabolic, have vanishing higher cohomology. Next we prove a simple but important lemma, which makes it possible to use induction as a tool. As a first application a description of the global sections of any line bundle is given. Then we show that the vanishing of Hl already implies the vanishing of Hi for all i 2: 1. Next we give an algorithm for calculating the Euler characteristic and two bounds on the order of vanishing of higher cohomology. Finally we give the normality result of some nilpotent varieties.

In another article we hope to give applications for modules of any sheaf of twisted differential operators on G / B .

1. Notation

We fix the following notation. G is a connected, complex semisimple linear algebraic group of rank T. B is a Borel subgroup containing the maximal torus T; we denote the Lie algebra of any algebraic group by the corre­sponding gothic character. We denote the cotangent bundle T* G / B of G/B by X.

B determines a set of positive roots R+ and dominant weights X+

Normality of Some Nilpotent Varieties 3

in the character group X = X* (T) of T. The Chevalley partial order on X is denoted by :5; i.e., A :5 I-' iff I-' - A is in the monoid generated by R+. For A = E;=l nilli, where lli are the simple roots, the height of A is ht A := E;=l ni. We choose for any A E X+ a simple highest weight module VA; m~ is the multiplicity of the weight I-' in VA'

Write W for the Weyl group, we fix a W-invariant inner product (,) on X ®z Q. If X E X write X+ for the dominant weight in the Weyl group orbit of X.

Let P be a parabolic subgroup. For a P-module V we write G x P V for the homogeneous G-vector bundle on G j P with fibre V above P E

GjP (see [3]); we write CG/p(V) for the locally free OG/p-module oflocal sections. If V' is another P-module, pulling back gives a G-vector bundle G x P (V x V') on G x P V with sheaf of sections CGXpv(V'). If V is completely reducible, the cohomology groups Hi(GjP,CG/p(V)) can be computed by the theorem of Borel-Weil-Bott. The cohomology groups of CGXpv(V') have a natural structure of a graded G-module by

00

Hi (G x P V,CGXPy(V')) ~ EBHi (GjP,CG/p(SiV* ® V')), i=O

where SiV* is the j-th symmetric power of the dual of V. If C = $iCi is any graded object we write Cli) for the shifted graded

object with Cli)i := CHi'

2. Line bundles on T*GjP

In this section we generalize some results in [5) on the cohomology of line bundles on T*GjB to line bundles on T*GjP, for parabolic subgroups P. Also the question is considered when the module of global sections has the interpretation of ideal.

2.1. A vanishing result. Let P :J B be a parabolic subgroup. Its cotan­gent bundle T*GjP can be identified with G x P (gjp)* and the moment map associated to the natural symplectic structure can be identified with the collapsing

I-' : G x P (gjp)* ~ g*: 9 * x 1--+ Ad*(g)x,

where Ad* is the coadjoint representation. Generically its fibres are fi­nite. R. Elkik observed that the vanishing theorem of H. Grauert and O. Riemenschneider can be applied to show that the higher cohomology of the structure sheaf vanish; this in turn implies (using a result J.-F. Boutot)

4 Bram Broer

that the normalization of the image of the moment map has rational sin­gularities, (see [13, Proposition 10.2]). The following is a generalization of a part of [5, Theorem 2.4), where the special case P = B is handled with a different proof. Its proof uses yet one more variation of Elkik's observation.

For a parabolic subgroup P :J B we write M(P) for the monoid of dom­inant weights A E X+ such that P stabilizes a one dimensional subspace in V>..

Theorem 2.2. Let P :J B be a parabolic subgroup. For A E M(P) we have for i 2': 1

Proof. Consider U:= G x P «g/p)* x C>.). It contains T*GIP and has a collapsing to g* x V>.:

¢: U ~ g* x V>.: 9 * (X, x) I--> (Ad*(g)X,gx).

This map is proper, since GIP is projective and ¢ factorizes as

hence has a closed image. The fibre of (x, 0) is contained in G x P «g/p)* x {O}) ~ T*G I P, and consists for generic x of finitely many points. Hence the same property holds for a generic element in the image of ¢.

Since U is the total space of a G-linearized vector bundle on G I P, the canonical bundle Wu is a pull-back line bundle from G I P. Since the top exterior power of the cotangent space in e * x (with e the identity in G, and x E (g/p)* x C>.) is

we have Wu ~ .cu(C>.)*. Now we can apply the theorem of Grauert­Riemenschneider in Kempf's version (see [10, Theorem 4)), to get

Since U is a vector bundle on G I P and Wu a pull-back from G I P, we

Normality of Some Nilpotent Varieties 5

have the isomorphisms

00

~ EBHi (GIP,£G/p (Cl ® sj «gil') EEl cl»)) j=O 00 00

~ EBEBHi (GIP,£G/p (Cl ®sj (gil') ®sn (cl»)) n=Oj=O

00

~ EBHi (T*GIP,£ToG/P (C(n+1)>')*)' n=O

It follows in particular that

This finishes the proof. • 2.3. Global sections as ideals. For some>. E M(P) the module of global sections of the coresponding line bundle £To G / P (C >.) * has an interpretation as ideal in C[T*G I Pl. In the special case of P = B this always is the case, and these interesting ideals were studied in [5] and [6].

Use the Killing form to identify (g/p)* with n, the nilradical of p. Let L be a Levi factor of P. We define an additive map htp : X -+ Q by prescribing the values on the simple roots. Let a be a simple root; if a is a weight of [ put htp(a) := 0, otherwise put htp(a) := 1. Then htB coincides with the ordinary ht.

Proposition 2.4. Let P be pambolic and>' E M(P). Suppose there exists a P-equivariant map a : n -+ C>. homogeneous of degree j. Then j = htp(>.) and HO(T*GI P, £ToG/ p(C>.)*)[- ht(>.)] is isomorphic to the gmded ideal of C[T*GIP] genemted by a basis of the unique copy of v; in the homogeneous part of degree ht(>.). This ideal defines the complement of the open orbit in Spec C[T* G I Pl·

Proof. The P-equivariant map a induces a section s of the line bundle £ToG/P(C>.), with some zero scheme Z(s), say. The dual of the section induces a short exact sequence of graded sheafs on Y := T*GIP

where L : Z(s) -+ Y is the inclusion map. So H°(Y, £y(C>.)*)[-j] C C[Y] is an ideal. Even more holds. By the vanishing Theorem 2.2 the following

6 Bmm Broer

sequence is exact

0-+ HO(Y,.cy(CA)*)[-jj-+ C[Yj-+ C[Z(s)j-+ 0,

and Hi(Z(s),OZ(s») = 0 for i ~ 1. In this way H°(Y,.cy(CA)*)[-j) acquires the interpretation of an ideal.

The map 0' also gives rise to a G-equivariant map

with image the highest weight variety of VA' With this covariant, the zero scheme Z(s) can also be described as the schematic fibre of the origin. Hence the ideal sheaf is generated by global regular functions forming a module isomorphic to V;. Therefore, the ideal of C[Z(s)) is generated by its degree j elements, spanning a module isomorphic to V;.

htp induces a graded Lie algebra structure on g, p and n, and a graded module structure on any ,,-module. By results due to Richardson, L has an open orbit in nlo say Le, and Pe is dense in n. Any G-covariant T*G / P -+

VA homogeneous of degree j is completely determined by the image of e in (VA)j, so j is at most htp(A). If j = htp(A) then the covariant is unique upto a scalar. The covariant constructed above maps e into CA C VA, so is essentially unique of degree ht(A). The image of this covariant is in the highest weight variety of VA' From this the proof of the proposition follows.

2.5 Remark. The description of the possible A'S that can occur in the proposition is equivalent to a description of the graded subring of C[n) of the P-semi-invariants. It is a polynomial ring of s ~ r generators of cer­tain degree-weight combinations (db Al)," ., (ds, As); this is a result due to Sato and Kimura (see [14, p. 202]). Let's give a geometric interpretation of the generators. The complement n\Pe (see proof above) is isomorphic to nl \Le; the simple roots of htp one are just the lowest weight of the L-module nl' Let Db"" Ds be the irreducible hypersurfaces in the com­plement of Pe (if there are any), their defining polynomials It, .. . , fs can be taken as generators of the ring of semi-invariants.

2.6. Example. Let G be of type G2 and P the parabolic subgroup such that minus the short simple root is a weight of p. For each i ~ 1 there is a P-covariant n -+ C2iw2 homogeneous of degree 4i, and these are the only ones, W2 is the longest root. Here C [T* G / Pj is isomorphic to the coordinate ring of the lO-dimensional normal subregular nilpotent variety (see [15)). The ideals HO(T*G/P,.cT *G/p(C[O,2ij)*)[-2jj all define the 8-dimensional non-normal nilpotent orbit closure.

For the other maximal parabolic subgroup P :::) B there is no P-stable codimension one subscheme of n.

Normality of Some Nilpotent Varieties 7

3. Cohomology of line bundles on T*G/B

In this section we shall mainly be concerned with line bundles on the cotan­gent bundle of the full flag variety. The main tool is some kind of induction on weights. The next lemma will be needed.

Lemma 3.1. Let Q ::) P be two pambolic subgroups and let V be an irreducible P-module. Write Z:= G x P (g/q)*.

(i) There exists at most one i ~ 0 such that

(ii) If Hi(Q/P,CQ/p(V)*) = 0 for all i ~ 0, then for all i ~ 0

Hi(Z, Cz(V)*) = o.

(iii) Suppose V:= HV(Q/P,CQ/p(V)*) =I 0 for v ~ O. Then

H i(Z C (V)*) {O ifi < Vj ,z = Hi-v(Z, Cz<V)*) ifi ~ v.

Proof. (i) is a consequence of the Borel-Weil-Bott theorem. For all j ~ 0 there is a Leray spectral sequence associated to the projection G / P --+ G /Q with fibres isomorphic to Q/P, with second term

E~I = Hk (G/Q,CG/Q (HI (Q/P,CQ/p(Si g/ q 0 V*»)))

= Hk (G/Q, CG/Q (Sig/q 0 HI (Q/P,CQ/p(V*»)))

=> Hk+I(G/p,CG/p(Sig/q 0 V*».

By (i) this sequence degenerates, implying that

Now (ii) follows immediately and the vanishing of cohomology in (iii) if i < v. The remaining statement follows from

• 3.2. The induction lemma. Let 0: be a simple root and Q ::) B the associated minimal parabolic subgroup, with

(g/b)* /(g/q)* ~ (q/b)* ~ Ca. (1)

8 Bmm Broer

The line bundle .c x (COt) on the cotangent bundle X of G j B has a natural linear section with scheme of zeros Z = G x B (gjq)*; write t : Z C X for the inclusion. We can identify .cx (COt)*[-I] with the ideal sheaf of Z in X, where the [-1] denotes a shift in grading such that generators have degree 1. So we get for every >. E X a G-equivariant exact sequence of graded Ox-modules

For>. E X(T) we shall write

These are graded modules over qN], with compatible rational G-module structure, where we identified qX] with the coordinate ring of the full nilpotent cone N.

Lemma 3.3. Let>. E X and let a be a simple root such that S>' - a = na for some n ;::: 1, where s := SOt is the reflection associated to the simple root a.

(i) We have Hg(>.) = 0 and HO(>.) ~ HO(>. + a)[-I].

(ii) For all i ;::: 1 we have H~(>') ~ H~-l(S>' - a).

(iii) If n = 1, then H~(>') = 0 for all i ;::: 0, and so Hi(>.) ~ Hi(s>')[-I], for all i ;::: 1.

(iv) Suppose Hk(>. + a) = Hk-1(s>. - a) = Hk(s>.) = 0, then Hk(>.) = O.

Proof. In this case Qj B is the one dimensional projective space, and

The statements about H~ ( -) follow from Lemma 3.1. The other state­ments follow easily using the long exact sequences associated to (1) for >. and S>' - a:

••. ---+ H~-l(>.) ---+ Hk(>. + a)[-I] ---+ Hk(>.) ---+

II ... ---+ H~-2(S>' - a) ---+ Hk-l(S>.)[-I]---+ Hk-1(s>. - a) ---+

(2)

• 3.4. Description global sections. As a first application of the induction lemma we shall describe the global sections of any line bundle on T* G j B

Normality of Some Nilpotent Varieties 9

in terms of Brylinski's twisted ideals of the nullcone. We first recall the special case for dominant weights, Proposition 2.4. For more information see [5] and [6].

Theorem 3.5. Let A and Jl be dominant weights. Then Jl :::; A if and only if there is a non-zero G-equivariant homomorphism HO(A) ~ HO(Jl) of C[N]-modules.

Proof. In [5] the following facts are shown. (i) HO(A) has a generating set as a C[N] module spanning a simple G-module V;. (ii) If V; occurs in HO(A) then A :::; v. (iii) If Jl :::; A there exists a non-zero homomorphism HO(A) ~ HO(Jl).

If Jl 1, A there is therefore no submodule V; c HO(Jl), which implies that there is no non-zero G-equivariant C[N]-homomorphism HO(A) ~ HO(Jl). This proves (i), and (ii) follows easily. •

3.6. For a weight A E X we define A * to be the unique dominant weight that is minimal with the property that A * ? A. That it is well-defined follows from the proof of the next theorem stating that the space of global sections of any line bundle on T* G / B is a twisted ideal of the nullcone.

Theorem 3.7. Let A E X. There is a G-equivariant isomorphism

of graded C[N] -modules.

Proof. Using Lemma 3.3(i) it is easy to show by induction that there is a dominant weight Jl ? A such that HO(A) ~ HO(Jl)[- ht(Jl- A)]; by 3.5 this weight Jl is unique.

As in [5, Corollary 2.2] one can show that for any dominant weight X ? A there is a unique copy of V; in HO(A) in the homogeneous component of degree ht(X - A). One can now finish the proof using this remark and the facts cited in the proof of 3.5. •

3.S. Higher order vanishing. Write Cht(A) for the combinatorial di­mension of the interval [A *, A +] C X+ in the Chevalley order, i.e., the supremum over all r such that there exists a chain

A * :::; Jlo < Jll < ... < Jlr :::; A +,

with all Jli dominant.

10 Bmm Broer

In [5] we showed that .cx{,x)* has higher order vanishing of cohomology if and only if ,x + = ,x *. We shall give a new proof and show that there is higher vanishing if and only if HI (,x) = O. Our interest in this strengthening comes from [6, Hypothesis 6.6]. It raises the question: if Hi (,x) = 0 does it follow that Hi (,x) = 0 for j ~ i?

The vanishing theorem Hi(,x) = 0 if i > Cht(,x) will turn out to very useful for the proof of normality for some nilpotent varieties. It is not sharp in general.

Theorem 3.9. (i) For,x E X we have the equivalences

Hi(,x) = 0, for all i ~ 1 ¢:} HI(,x) = 0 ¢:} Cht(,x) = 0, i.e.,,x* = ,x+.

(ii) If Cht(,x) = 1, then up to a shift in degrees

(iii) Hi(,x) = 0, for i > Cht(,x).

Proof. We shall first show (iii) by an induction. If,x is dominant then Cht(,x) = 0 and by Theorem 2.2 we have higher vanishing. Now suppose ,x is not dominant and that the statement in (iii) is correct for J.L such that J.L+ < ,x + and for J.L such that J.L > ,x and J.L+ = ,x + .

Let a be a simple root such that sQ(,x)-,x = n,x with n strictly positive. Put m := n/2 if n is even and m := (n + 1)/2 if n is odd and write s := SQ. Then up to shifts in grading

And so by Theorems 3.5{i) and 3.7

,x* = (,x+a)* = ... = {,x + ma)* ~ {,x + (m+ 1)a)* ~ ... ~ s,x*.

We remark that the inequalities need not be strict. And of course

It follows that

Cht(,x) ~ Cht(,x + a) ~ Cht(s,x - a)

and Cht(,x) ~ Cht(s,x).

Normality of Some Nilpotent Varieties 11

If n = 1 above, then H i (>.) ~ Hi(S)..), by Lemma 3.3(i), and Cht()..) = Cht(s)..). So we can assume that n ~ 2. We now have that Cht(s)..) ~ Cht()..) and

Cht(s).. - a) ~ Cht().. + a) < Cht()..),

where the last inequality is strict. So by the induction hypothesis we have for i > Cht()..)

By Lemma 3.3(iii) it follows that Hi()..) = 0 for i > Cht()..). This proves (iii). The implication (3) =? (1) in (i) is a corollary and (1) =? (2) is obvious.

To show (2) =? (3) we again use induction. If).. is dominant then Hi()..) = 0 for i ~ 1. Take).. E X with Hl()..) = 0, and suppose that the implication (2) =? (3) is correct for every Jl such that Jl+ < ).. + and for every Jl such that Jl > ).. and Jl+ = ).. +. Let a be simple as above. By Lemma 3.3 there is an injection Hl().. + a) -+ Hl()..), so Hl().. + a) = O. Since Cht()..+a) = 0 by hypothesis we get Cht(s)..-a) = o. It then follows by (iii) that

H~()") ~ H~(s)" - a) = O.

First of all this implies that Hl(S)..) = 0 and by the induction hypothesis Cht(s)..) = O. And next that HO(s)..) ~ HO(s).. - a). So by Theorem 3.7 we get

s)..* = (s)..-a)* = (s)..- a)+ = ()..+a)+ = ()..+a)* = )..*.

So Cht()..) = Cht(s)..) = o. This finishes the proof of (i) Now suppose Cht(A) = 1 and a as above with n ~ 1. Then necessarily

n = 1,2 or 3. If n = 1 then Cht(sA) = 1 and Hl(A) = Hl(SA)[-I], and by induction the statement holds for SA, then also for A.

Next suppose n = 2 or 3, then we have Cht(A + a) = Cht(sA - a) = O. And

A* = (SA -a)* = (sA-a)+ < A+.

Suppose first that Cht(s)..) = O. Then just as earlier in this proof we have

12 Bram Broer

that

H 1(>,) ~ H~(>") ~ H~(s>" - a)

~ HO(s>.. - a)/Ho(s>..)[-I]

~ HO(>"*)[- ht(>..* - s>.. - a)]/ HO(>..+)[- ht(>..+ - s>..) - 1]

~ HO(>..*)[- ht(>..* - >..) + n - III HO(>..+)[- ht(>..+ - >..) + n - 1].

Next suppose that Cht(s>..) = 1. By Lemma 3.3 again there is an isomorphism

Now (ii) follows by induction. • 3.10. Remarks. (1) The shift in degree in (ii), say by 0, can be described by Lusztig's q-analog of weight multiplicity M~* = qht(A*-A) - q6 (see [5, Proposition 2.1]).

(2) Lemma 3.3(iv) suggests a procedure for obtaining another bound on the order of vanishing of higher cohomology. Let us give the (inductive) definition first of some integer valued function N(>") on X. For>.. dominant put N(>") := OJ otherwise suppose N(Jl) has been defined for weights Jl such that >.. < Jl ~ Jl+ ~ >.. +. Let a1, . .. ,al be all the simple roots such that sai(>") - >.. = ni>" with ni ~ 1. Write Ni := N(sai(>")) if ni = 1 and Ni := Max{N(>.. + ai), N(sa; (>..) - ai) + 1, N(sa; (>..))} if ni > 1. Now put N(>") := Min{Ni , 1 ~ i ~ I}. From Lemma 3.3(iv) it follows that this number provides a bound for higher vanishing of cohomology:

Hi(>..) = 0, for i > N(>").

Unfortunately we cannot prove much for N(>"), we don't know how good it is in general as a bound on higher vanishing. For the rank ~ 2 cases we believe it is sharp. Apart from that, we would rather have a priori estimates. If w(>.. + p) - >.. is dominant is Hi(>..) = 0 for i bigger than the length I of w? Is N(>") = l?

3.11. Lemma on Cht. The following lemma will be used in the proof of normality of some nilpotent varieties.

Lemma 3.12. Let G be simple and let a1, a2,' .. ,an be pairwise perpen­dicular short simple roots, where n ~ 1. Put a := a1 + ... + an. Then a*

is the short dominant root ¢ and Cht( a) = n - 1.

Proof. For any i we have ai ~ ¢; since ¢ can be expressed uniquely as a sum of simple roots and ¢ is the smallest non zero dominant weight in the

Normality of Some Nilpotent Varieties 13

root lattice it follows that a* = cpo The statement about Cht follows from the next combinatorial lemma. •

Lemma 3.13. Let..\ E X and (3 a short root orthogonal to..\. Then..\+ < (..\ + (3)+, and for a dominant weight I" the inequalities ..\ + :5 I" :5 (..\ + (3)+ imply that I" =..\+ or I" = (..\ + (3)+.

Proof. The first statement follows from

To show the second statement we can first of all assume that ..\ = ..\ + is dominant, since the Weyl group preserves the inner product, and next that (3 is positive, since (..\ + (3)+ = (s/3(..\»+ = (..\ - (3)+.

Let I" be dominant such that ..\ < jL, then

11"12 = 1..\12 + 1(1"- ..\)12 + 2(,,\, (1"- ..\»)

~ 1..\12 + 1(312,

since (I" - ..\) is positive and ..\ dominant and since (3 is short. Equality holds if and only if (I" - ..\) is a short positive root and perpendicular to ..\. In particular, there is a short positive root (3' perpendicular to ..\, such that (..\ + (3)+ = ..\ + (3'. Let (3" be any positive root such that ..\ + (3" is dominant and ..\ < ..\ + (3" :5 ..\ + (3'. Then

1..\12 + 1(312 = 1..\ + (3'12

= 1..\ + (3"12 + 1(3' - (3"12 + 2(,,\ + (3", (3' - (3")

~ 1..\ + (3"12

~ 1..\12 + 1(312.

It follows that (3' = (3". • 3.13. Euler characteristic. Although the induction lemma is not strong enough to calculate all the higher cohomology groups, it does pro­vide an algorithm to calculate the Euler characteristic in the appropri­ate Grothendieck group. Write KGxc. (N)o for the Grothendieck group of finitely genemted gmded C [N] -modules with a compatible mtional G -module structure. So if

o -+ M' -+ M -+ M" -+ 0

14 Bram Broer

is a G-equivariant short exact sequence of graded IC[N'j-modules with com­patible rational G-module structure, then

[M) = [M') + [M")

in KGxc. (N)o, where [M) denotes the class of M. This Grothendieck group has a natural Z[q, q-l)-structure induced by qi . [M) .- [M[-ill, where M[-i) is the shifted module.

For>. E X put

Eul(>.) := L) _1)i[Hi(>.)] E KGxc· (N)o , i

for the Euler characteristic of .cT. G / B (C >..) *. It can be calculated ind uc­tively using the next result.

Proposition 3.15. Let>. E X and 0 a simple root such that sa (>.)->. = no for n ;:::: 1. Then we have

Eul(>.) + Eul(s,,(>.) - 0) = q (EuI(>' + 0) + Eul(s,,(>.))).

Proof. This follows easily from the induction lemma, the two long exact sequences (2) and that L:i(-1)i[H~(>')] = - L:(_1)i[H~(sa(>') - 0)]. •

3.16. Remarks. It follows immediately that Eul(>.) is in the Z[q, q-l]_ submodule of KGxc.(N)o generated by the HO(Il), with Il E X+. More generally, let :F be any G-equivariant graded coherent sheaf of OT" G / B­

modules. Since G is reductive, :F has a finite resolution by locally free G-equivariant graded modules. Since the .c(CIl ) are the simple locally free G-modules on T*GIB, any graded locally free module has a filtration with subquotients isomorphic to shifted .c(CIl ) [sl's. It follows then that the Euler characteristic of :F can be expressed in terms of the classes HO(Il), where Il is dominant.

This is also the case for the normalization of the coordinate ring of any nilpotent variety N C N. A proof of this runs as follows. There is a desingularization by the collapsing of a vector bundle G x P 't-> N, for some Lie subalgebra 't C p. By a result due to D. Panyushev [16] and inde­pendently V. Hinich [9] the normalization of N has rational singularities. But then the collapsing G x B 't->N has higher vanishing cohomology and HO(C x B 't, 0GxBr) can be identified with the normalization of IC[N].

Normality of Some Nilpotent Varieties 15

generated graded C[N]-module M = ffiiMi with compatible G-structure, and for >. E X+ we put

MA(M):= LIMi : V;lqi i

for the >.-multiplicity (Laurent) polynomial of M. We get a Z[q, q-l]-linear map

We put M~ := MA(Eul(Jl»; this is just Lusztig's q-analog of weight mul­tiplicity (see [5]). The specialization M~(q := 1) equals the T-weight mul­tiplicity m~ of Jl in VA. It identifies with a Kazhdan-Lusztig polynomial of the associated affine Hecke algebra if Jl E X+.

To any finitely generated C[N]-module M we can associate its gener­ating function

1i(M) = L dim(Mi)qi. i

T his induces a Z[q,q-l]-linear map 1i: KGxc.(N)o -+ Z[q][q-l]. The order of the pole in q = 1 is the dimension of the support of M; also the multiplicity can be read of. The support of HO(>.) is N; the dimension and multiplicity of HO(>.) and HO(>.*) are equal. For i ;:::: 1 the support of Hi(>.) support is strictly smaller than N of even dimension. For i = dim( G I B) it is supported on the origin, i.e., is a finite dimensional vector space. It would be interesting to know the support of Hi(>.) in general; is it irreducible?

4. Normality of some nilpotent varieties

Let G be semisimple and P a parabolic subgroup. Write ¢p: T*GIP-+ g* ~ 9 for the moment map followed by the identification of g* with 9 using the Killing form. Then ¢p is a proper map with image a nilpotent variety Gn in g, with n the nilradical of p; in fact it is the closure of the Richardson nilpotent orbit. In general ¢p need not be birational with normal image.

Theorem 4.1. Let G be semisimple. Let P ::) B be a parabolic subgroup such that the weights of p/b are pairwise orthogonal and short. Then ¢ is birational onto its image Gn. This image Gn is an affine Gorenstein nilpotent variety with rational singularities, which implies that it is normal.

16 Bram Broer

Proof. Write n for the nilpotent radical of p and

(p/b)* =: Cal 61 ... 61 Co"'

where by assumption the occurring roots are short and mutually orthogo­nal; they are all simple. The direct sum of line bundles on X = T*G / B

has a natural linear section with zero scheme t : Z := G x B (O/p)* C X. We get a locally free Koszul resolution

0- "n.cx(p/b)[-nj_ ... - ,,2.cx(p/b)[-2j

- .cx(p/b)[-lj- Ox - t*Oz - 0, (3)

of t*Oz, which preserves grading and is G-equivariant. For every l, "l.cx(p/b) is a direct sum of line bundles .cx(C,.)*, where J.L is the sum of 1 of the mutually orthogonal simple roots Q:i; by Lemma 3.12 Cht(/-l) = l- 1 and so Hj(J.L) = 0 for j ~ l. Therefore Hj(X, "l.cx(p/b» = 0 for j ~ l. Let I be the ideal sheaf of Z. By breaking up the Koszul complex in short exact sequences, one can now easily show that HI(X,I) = 0 if 1 2: 1. There results a short exact sequence

and Hi(Z,OZ) = 0 for i ~ 1. By Lemma 3.1 the surjection 7r : Z - T*G/ P induces Hi(Z, Oz) ~

Hi(T*G/P,OG/p) for all i. In particular, we have a surjection f : C[NJ-C[T*G/Pj. The restriction to Z of ¢B equals 7r followed by ¢p. The Stein factorization of ¢p induces a factorization of f:

c[NJ-C[Gnj C C[T*G/Pj.

The surjectivity of f implies that C[Gn] = C[T*G/Pj. Since T*G/P is smooth C[T* G / Pj and Gn are normal. The birationality of ¢ p and that Gn has rational singularities follow now easily. Since the canonical bundle of the symplectic variety T* G / P is trivial, the canonical bundle on Gn is trivial; this implies the Gorenstein property. •

4.2. Examples. This theorem covers the full nilpotent cone (Kostant), the subregular nilpotent variety (see [5]) and a handful of others for each simple group. For SL(r + 1) these are the closures of the orbits with at

Normality of Some Nilpotent Varieties 17

most two blocks in their Jordan normal form. We get new results for the exceptional simple groups. We use the tables and terminology of Carter's book [7].

For E6 we get the normality for E6(a3) and Ds; the normality of the latter was already shown in my thesis [4, p. 85] by a different method, using results of Richardson.

For E7 we get normality for E6 (al), E6 , E7(a3) and E7(a2)j for the latter (see again [4, p. 85]).

For Es we get normality for ES(a4), ES(a3) and Es(a2)'

4.:1. Graded C-structure. The coordinate ring C[Cn] is graded and has a rational C-module structure.

Corollary 3.4. Let C be semisimple. Let P ~ B be a parabolic subgroup such that the weights 01."" Om of p/b are mutually perpendicular and short. (i) We have

M),(C[Cn]) = Jc{1 •...• m}

where OJ = Ojl + ... + Ojq if J = {it, ... ,jq} and where M~ is Lusztig's q-analog of weight multiplicity (see remark 3.17).

(ii) The multiplicity of v; in C[Cn] equals

L (-1)lJlm~J, Jc{1 ..... m}

where m~ is the T-weight multiplicity of I" in V),. It is also the dimension of the fixed point space in V; of the Levi factor L of P. Hence C[Cn] is C-isomorphic to C[C/L].

Proof. This follows easily from the theorem and the exact sequence (3). See the remarks in 3.17. The theorem implies (ii) in another way by using the associated cone construction of Borho-Kraft [3]. •

4.5. Example. We finish with an example for a class in Es, denoted ES(a2)' Take 04 and 06 in the Bourbaki enumeration of simple roots of Es, they define a parabolic subgroup P. Now (04 +(6)+ = WI. (04 +(6)* = Ws,

ht(ws) = 29, ht(Wl) = 46. If I is the ideal in C[N] of the closure of the Richardson orbit associated to P, then there is a short exact sequence

18 Bmm Broer

(to see this, use Theorem 3.9(ii». It follows that

References

[I] H.H. Andersen and J.C. Jantzen, Cohomology of induced representa­tions for algebraic groups, Math. Ann. 269 (1984), 487-525.

[2] A. Beilinson and J. Bernstein, Localisation de g-modules, C.R. Acad. Sci., Paris, 292 (1981), 15-18.

[3] W. Borho and H. Kraft, Uber Bahnen und deren Deformationen bei linearen Aktionen reduktiver Gruppen, Comment. Math. Helv. 54 (1979), 61-104.

[4] A. Broer, Hilbert series in invariant theory, Thesis, Rijksuniversiteit Utrecht (1990).

[5] A. Broer, Line bundles on the cotangent bundle of the flag variety, Invent. Math. 113 (1993), 1-20.

[6] R.K. Brylinski, Twisted ideals of the nullcone, In: A. Connes, M. Duflo, A. Joseph, R. Rentschler (Eds.) Operator algebras, unitary represen­tations, enveloping algebras and invariant theory, Birkhiiuser, Boston, 1990, 289-316.

[7] R.W. Carter, Finite groups of Lie type. Conjugacy classes and complex characters, Wiley, Chichester, 1985.

[8] W.H. Hesselink, Cohomology and the resolution of the nilpotent vari­ety, Math. Ann. 223 (1976), 249-252.

[9] V. Hinich, On the singularities of nilpotent orbits, Israel J. Math. 73 (1991), 297-308.

[10] G.R. Kempf, On the collapsing of homogeneous bundles, Invent. Math. 37 (1976), 229-239.

[11] B. Kostant, Lie group representations on polynomial rings, Amer. J. Math. 85 (1963), 327-404.

[12] H. Kraft and C. Procesi, Closures of conjugacy classes of matrices are normal, Invent. Math. 53 (1979), 227-247.

[13] H. Kraft and C. Procesi, On the geometry of conjugacy classes in clas­sical groups, Comment. Math. Helv. 57 (1982), 539-602.

[13] H. Kraft, Geometrische Methoden in der Invariantentheorie, Vieweg Verlag, Braunschweig, 1984.

[14] H. Kraft, Closures of conjugacy classes in G2 , J. Algebra 126 (1989), 454-465.

Normality of Some Nilpotent Varieties 19

[16] D.l. Panyushev, Rationality of singularities and the Gorenstein prop­erty for nilpotent orbits, Funct. An. Appl. 25 (1991), 225-226.

[17] J. Weyman, The equations of conjugacy classes of nilpotent matrices, Invent. Math. 98 (1989), 229-245.

Mathematics Department, The University of British Columbia, Vancouver, B.C.,

Canada V6T 1Z2.

Current address: Mathematics Department, University of Montreal, Montreal,

QC, Canada H3C 3J7.

Received March 28, 1994

Holomorphic Quantization and Unitary Representations of the Teichmiiller Group

Jean-Luc Brylinski and Dennis McLaughlin*

Dedicated to Bert Kostant with admiration

One of the purposes of geometric quantization (in the sense of Kostant) is to associate in some intrinsic way a finite-dimensional Hilbert space E to a compact symplectic manifold (M, w) such that 27fH . w is the cur­vature of a connection '\7 on some line bundle L. It is usually assumed that M is simply-connected, since then the pair (L, '\7) is unique up to iso­morphism, as was shown in [Ko]. In case there is a finite-dimensional Lie group G of symplectomorphisms present, a central extension G of G was constructed by Kostant [Ko] , and this central extension should act on E. Thus Kostant was able to recover the Borel-Weil-Bott theorem from the action of a compact Lie group G on a quantizable coadjoint orbit.

The construction of E requires a polarization of the symplectic mani­fold M, that is to say a foliation with lagrangian leaves; this polarization could be real, or complex as in the case of a coadjoint orbit of a compact Lie group. One nice type of polarization is that induced from a Kaehler structure: the polarization is given by the antiholomorphic component of the complexified tangent bundle. If (M,w) is Kaehler, then L becomes a holomorphic line bundle, and one can take E to be the space of global holomorphic sections of L (which is finite-dimensional by a theorem of Serre [Serl)o If G preserves the Kaehler structure (which implies that G is compact), then a central extension of G will act on E. However, in some cases G can be made to act on E even though its action does not preserve the complex structure of M; a classical example is the action of a com­plex semisimple Lie group on its flag manifold. This method can be called holomorphic quantization. Much more subtle examples (for non compact symplectic manifolds) have been found in [Br-Ko] for nilpotent orbits of a semisimple Lie group Ge which only admit a Ke-invariant polarization, where Ke is the complexification of a maximal compact subgroup of a real form G of Ge.

* Each author was supported in part by a grant from the NSF.

22 J.-L. Brylinski and D. McLaughlin

Witten's field theoretic treatment of the Jones polynomial [Wi] has focused attention on geometric quantization applied to the moduli space B of flat G-bundles over a compact surface E of genus g, for G a compact simple Lie group. Atiyah and Bott proved that B admits a quantizable symplectic form w [A-B]; although B is not compact, it has a compactifi­cation for which the boundary has large codimension, so that it can often be treated as if it were compact. One obtains a Kaehler structure on B for any choice of a complex structure on E, hence we have a Kaehler manifold B(E, Gc) associated to any Riemann surface E. Then one has a holomorphic line bundle Lover B, and one can define the vector space E to be the space of global holomorphic sections of L®k, which is indeed finite-dimensional. The vector space E, thus attached to B(E, Gc), has its dimension given by the Verlinde formula [V], for which there are now rigorous proofs [B-L] [B-Sz] [D-W] [F2] [N-R) [Sz] [Th) [J-K) [T-V-Y].

This leads to a typical problem of geometric quantization: There is a natural symplectic action of the Teichmiiller group r 9 (the group of outer automorphisms of 1I'1(E)) on the symplectic manifold B, but this action does not preserve any of the Kaehler structures. The question is then to define a natural action of (a central extension of) r 9 on E. This has been done by various authors [Seg) [A-DP-W) [K-N-T-Y) [Hi) [Fl] [B­G] [Be-Ka). All these authors follow a geometric method suggested by Witten [Wi), which consists of constructing an integrable connection on a vector bundle £ over the moduli space of Riemann surfaces of genus g,

with fiber at E equal to the space E above. Actually the connection is only projectively integrable, i.e., its curvature is a scalar valued 2-form. Then the monodromy of this integrable connection gives rise to a representation of a central extension of r g.

In this paper we present two approaches to the integrable connection of Witten and its monodromy. The first, developed in §1-3, is a continuation of the first author's work on non-abelian theta functions [Brl). It is based on the description of the vector space E in terms of loop groups [K-N-R), and on the Sugawara construction, which realizes the Virasoro Lie algebra as quadratic elements in some completion of the enveloping algebra of a loop algebra (see [P-S] [K-R)). This use of the Sugawara construction makes the connection appear as some sort of heat equation, similar to the methods of [A-DP-W] [Be-Ka] [Hi] . The second method is based on a new approach to the construction and study of holomorphic line bundles with hermitian metrics by cohomological means, presented in §4. Here we follow the lead of Kostant [Ko] , who associates to a line bundle with connection on a manifold X, a Cech l-cocycle (gij, (Xi) with coefficients in the complex of sheaves ~x ~ £1\. This provides us with a cohomological

Unitary Representations of the Teichmiiller Group 23

method to construct holomorphic line bundles with hermitian metrics; we apply this to the construction of a metric on a determinant line bundle. The metric we construct has the same curvature as the Quillen metric, but we do not know if it coincides with it. Our construction of the metric proceeds in two steps: first we use the results of [Br-ML2] to find a characteristic class for a universal Gc-bundle (which exists at least locally); then we use a transgression (in a bundle whose fibers are Riemann surfaces) to derive the metric on the determinant line bundle.

The point of this second approach is that our cohomological construc­tion of a holomorphic line bundle with metric works over the product of Teichmiiller space with B, and local curvature computations (done in §6) show that the curvature lives entirely in the B-direction. Using this fact, we hope to be able to derive a projectively integrable connection for the vector bundle £: over Teichmiiller space, which admits a natural horizon­tal hermitian form, given by integration over B. This goal is however not accomplished in the present paper. The interest in constructing such a hermitian form preserved by the connection is that the projective repre­sentation of r 9 deduced from the integrable connection (see [A-DP-W]) would then be unitary.

We view our cohomological approach to holomorphic quantization as an alternate approach to the theory of the Quillen metric on determinant line bundles, which is usually based on the spectral analysis of the Laplace operator [Q). The possibility of such a geometric approach, bypassing the analysis, first arose in the work of Deligne [De2]. From a technical stand­point one main contribution of our work is the setup of holomorphic her­mitian Deligne cohomology, which we hope will become a standard tool in the study of holomorphic vector bundles and in arithmetic algebraic geometry. Only the first two holomorphic hermitian Deligne complexes of sheaves are considered here (in §3 and in §5). The general holomorphic her­mitian Deligne complexes and their transgression maps will be described in another article by the authors.

We give a rather detailed account here of the way transgression in holomorphic hermitian Deligne cohomology is used to produce a hermitian holomorphic vector bundle over Teichmiiller space. The construction of the degree four characteristic classes in that cohomology theory and of the transgression map are only sketched here, as they will be described fully in [Br-ML4).

It is a pleasure to thank Scott Axelrod for interesting correspondence and discussions, and Rahul Pandharipande for useful discussions about his work [Pal. We are very grateful to Gerd Faltings who read the preprint version of this paper and pointed out several errors.

24 J.-L. Brylinski and D. McLaughlin

The first author thanks Harvard University for its hospitality during the period this work was written.

O. Conventions and notations

Following Deligne, for p E Z, we denote by Z(P) the cyclic subgroup Z(p) = (2rrA)P. Z of C generated by (2rrA)P.

We will denote by AP(X) the complex vector space of smooth complex­valued p-forms on a smooth manifold X, and by AP(X)1R the real vector space of smooth real-valued p-forms. If X is a complex manifold, there is the decomposition AP(X) = Ei A(i,p-i)(X) of p-forms into types. The exterior differential d decomposes accordingly as d = d' +d", where d' maps A(i,j)(X) to A(i+l,j)(X), and d" maps A(i,j)(X) to A{i.i+1)(X).

The sheaf of complex-valued p-forms on X will be denoted by Ai-, the sheaf of real-valued p-forms by Ai-,JR'

In the second part of this paper we use notations from sheaf the­ory which we now explain. Standard references are [Go), [B-T) and [I)j [Br2) has a presentation of sheaf cohomology which includes Deligne co­homology and its smooth version. For F a sheaf of abelian groups on a space X, the sheaf cohomology groups HP(X, F) that we use can al­ways be viewed as the Cech cohomology groups with respect to a suit­able open covering U = (Ui)iEI, that is the cohomology of the complex CP(U, F) = TIio, ... ,ip F(Uio, .. ip ) with differential (j given by the usual for­mula [Go). If F is the constant sheaf, then the open covering U can be used to compute the sheaf cohomology of F if it is a good covering, i.e., all intersections Uio"' ip are empty or contractible. If F is a coherent sheaf on a complex manifold X, then one needs the intersections Uio"' ip to be Stein.

We will always associate to a double complex (Kee, dH, dv ) (with hor­izontal differential dH and vertical differential dv) its total complex KP = Ei K(i,p-i), with total differential D given by Dx = dHx + (_l)i. dv(x) for x E K(i,p-i) (this is the Koszul sign convention). This construction of the total complex applies to a double complex of groups, or sheaves, etc ...

For a complex of sheaves Fe on X, which is bounded below, and for an open covering U, we have the double complex K(p,q) = CP(U,Fq), whose horizontal differential is the Cech differential (j, and whose vertical differ­ential is induced by the differential of the complex of sheaves Fe. The Dech hypercohomology of the complex of sheaves Fe is then the cohomology of the total complex of this double complex. The hypercohomology groups HP(X, Fe) can be viewed as the Cech hypercohomology with respect to a suitable open covering. Given a complex of sheaves Fe, we have cohomol­ogy sheaves 1{Q(Fe). The hypercohomology of Fe can often be computed

Unitary Representations of the Teichmiiller Group 25

from the cohomology groups of these sheaves, by using the so-called hyper­cohomology spectral sequence:

which we will have occasion to use in some proofs. To make life more interesting, we will also have to consider hyperco­

homology groups for some double complexes of sheaves K-- i this means the hypercohomology of the corresponding total complex of sheaves. In practice this is computed as the total cohomology of the triple complex CP(U, K(q,r».

To be very clear, the cohomology of a triple complex A(p,q,r) , with differentials dl, d2 , d3 is the cohomology of the associated total complex, with total differential d1 + (-I)Pd2 + (-I)(p+q)d3 •

1. Line bundles over moduli spaces of stable bundles

Let E be a compact Riemann surface of genus g, and let Gre be a simple com­plex Lie group. Then for each integer k ~ 1, there is a finite-dimensional vector space E = Er;,k which is of great importance in algebraic geometry and in conformal field theory.

To describe the vector space Er;,k, we need to introduce the mod­uli space B(E, Gre) of stable holomorphic Gre-bundles over E. Recall that according to [R] a principal Gre-bundle P -+ E is called stable (resp. semi­stable) if the associated Lie algebra bundle {I -+ E has the following prop­erty: for any parabolic Lie algebra subbundle p C (I, we have: deg(p) < 0 (resp. deg(p) ~ 0). The bundle p is called parabolic if for each x E E the fiber Px is a parabolic subalgebra of (Ix.

Recall the topological classification of principal Gre-bundles over E: ac­cording to [R], they correspond to elements of 71"1 (Ge). The correspondence is obtained by trivializing the bundle over the complement of a point p of E and over a little disk D centered at Pi then the transition function over the punctured disk D* gives an element of 71"1 (Gre).

We will assume that G is simply-connected, in which case the moduli space B(E, Gre) is connected [R, Prop. 4.2]. We will discuss the question of finding a local holomorphic Poincare Gre-bundle over an open subset U of B(E, Gre). By a Poincare bundle we mean a holomorphic Gre-bundle P -+ U x E such that, for every y E U, the Gre-bundle over {y} x E is isomorphic to the stable Gre-bundle corresponding to the point y of U ~ B(E, Gre). Here are some of the known facts about Poincare bundles:

(1) every point Y E B(E, Gre) has a neighborhood over which there

26 J.-L. Brylinski and D. McLaughlin

exists a Poincare bundle (this follows from [R, Theorem 4.2]); (2) any two Poincare bundles over a contractible open subset of

B(E, Gc) are isomorphic; (3) for a Poincare bundle P -+ U x E, the sheaf Aut(P) over U of

automorphisms of P is isomorphic to the constant sheaf Z(Gc). Here Z(Gc) is the center of Gc.

These properties, together with the obvious facts that Poincare bundles can be restricted to open subsets and satisfy the usual glueing properties of bundles, mean that we have a gerbe C (in the sense of Giraud [Gi]) over B(E, Gc) with band equal to the constant sheaf Z(Gc). The gerbe C associates to any open set U the category of Poincare bundles over U; there are natural restriction functors to smaller open sets. The obstruction to finding a global Poincare bundle is then a class in H 2(B(E, Gc), Z(Gc)), namely the cohomology class of the gerbe C. Let us recall how this class is constructed in tech cohomology. Take an open covering (Ui ) of B(E, C) such that there is a Poincare bundle Pi over Ui . Let ¢ij : (Pj)/u;j':::"'(Pi)/u;j be an isomorphism of Poincare bundles over Uij = Ui n Uj . Then the obstruction to glueing the Pi to a global Poincare bundle is given by the tech 2-cocycle K,ijk with values in Z(Gc), such that

where K,ijk is viewed as an automorphism of Pk.

The obstruction class in H 2(B(E, Gc), Z(Gc)) is sometimes referred to as the Brauer class [A-B]. It is not always trivial.

We now turn to the question of defining various determinant line bundles over B(E, Gc), associated to finite-dimensional representations Gc -+ Aut(V). If we have a Poincare bundle P -+ U x E over some open set U, then we have an associated vector bundle V = P x Gc V -+ U x E. Again there is an obstruction to constructing a global associated vec­tor bundle over B(±, Qc) x ±. The obstruction class is the image of [K,] E H2(B(±,Qc),Z(Qc)) in H2((B(±,Qc),0*) under the restriction of p to Z(Gc), which gives a character r : Z(Gc) -+ C*. Since the determi­nant of p is trivial, we have:

dim(V) . r([K,J) = 0 E H 2 (B(E, Gc), 0*). (1)

We will use this fact to define globally the determinant line bundle LV, a holomorphic line bundle over B(E, Gc). For any Poincare bundle over an open subset U of B(E, Gc}, we define LV over U to be the determinant line bundle of Knudsen and Mumford [K-M] with respect to the associated vector bundle V -+ U x E. Recall that the fiber of this determinant line

Unitary Representations of the Teichmiiller Group 27

bundle at y E U is the "determinant of the cohomology H*(E, V{Y}XE)",

that is

We claim that there is no obstruction to glue the local line bundles with respect to local Poincare bundles into a global line bundle over 8(E, Ge). Indeed a scalar automorphism A of V acts by

on the fiber of the determinant line bundle. The obstruction to globalize this line bundle is therefore equal to the class dim (V) . (1- g) . r(~), which is trivial by (1).

Proposition 1. [D-N] For any representation p : Ge -7 Aut(V), there exists a holomorphic line bundle £v over 8(E, Gd such that, if P -7 U x E is a Poincare line bundle over an open set U of 8(E, Ge), the restriction of £v to U identifies with the determinant line bundle of the associated vector bundle V -7 U x E.

Our proof of this is rather different from that of Drezet and Narasimhan.

It is easy to extend the construction of the determinant line bundle £v to any virtual representation V of Ge, in such a way that £VI-V2 ~

£Vl ® £~2-1 for any two representations V1 and V2 of Ge . This geometric method of constructing line bundles does not lead to

all holomorphic line bundles over 8(E, Ge). There is another method of constructing line bundles which is based on a group-theoretic description of the space Bund(E, Ge) of all isomorphism classes of algebraic Gre-bundles over E. This description is a version of that due to Pressley and Segal [P­S]. It depends on the choice of a point p of E and of a local parameter z at p, corresponding to an embedding of the closed disk D = {z E C : Izl ~ I} into E. Let O( D*) be the algebra of holomorphic functions on D* which are meromorphic at the origin; the algebra O(D*) contains two subalgebras:

- the subalgebra O(D) of holomorphic functions on D; - the subalgebra O(E \ {p}) of holomorphic functions on E \ p which

are meromorphic at x. The inclusion O(E \ {p}) ~ O(D*) is given by restriction of functions to D* c E \ {p}.

Let g = G(O(D*» and introduce the two subgroups r = G(O(D» and P = G(O(E \ {p}». Then we have, in analogy with [P-S]:

Proposition 2. The set Bund(E, Gd of isomorphism classes of holomor­phic Ge-bundles over E identifies with the double coset space r\Q /P.

28 J.-L. Brylinski and D. McLaughlin

Proof. It is well-known that every holomorphic Gte-bundle over D or over E \ {p} is trivial. These two open sets give a cover of E. Let 11" : P -+ E be a holomorphic Gte-bundle; let 81 (resp. 82) be a section of 11" over D (resp. over E \ {p}). Then we have a holomorphic function h : D* = D n (E \ {p}) -+ Gte such that 82 = 81 • h. If we change 81, we multiply h on the left by an element of r. If we change 82, we multiply h on the right by an element of P. Hence the double coset r· h· P depends only on P. This defines the map from Bund(E, Gc) to r\9 /P, and it is easy to see that it is a bijection. •

We will define the notion of holomorphic function on open subsets of r\9/p, following IB-L) IK-N-R), where those authors use the group G(C(z))) instead of G(O(D*». Recall that r\9 is the union of finite­dimensional complex algebraic varieties Xw, where Xw is the Bruhat cell r\r . w . r c r\Q associated to an element w of the affine Weyl group IP-S). The closure Xw is the (finite) union of the Bruhat cells X y , where y ::; w in the Bruhat order. We say that a complex-valued function f on an open subset V of r\Q is holomorphic if its restriction to each Schubert variety Xw is holomorphic. We say that a function f on an open subset of r\Q /P is holomorphic if the pull-back function on an open subset of r\9 is holomorphic. This defines a sheaf of algebras on the space r\9 /P. Let X be a complex-analytic variety and let f : X -+ r\Q /P be a continuous mapping. We say that f is holomorphic if for any holomorphic function f on an open subset V of r\Q/p, the pull-back function f*h on f- 1V C X is holomorphic.

The following lemma explains the significance of these notions in terms of families of Gte-bundles.

Lemma 1. Let X be a complex-analytic manifold, and let 11" : P -+ X x E be a holomorphic Gc-bundle. Let 1/1 : X -+ r\9 /P be the tautological mapping, such that 1/1(x) is the class of the Gc-bundle obtained by restricting P to {x} x E. Then 1/1 is a holomorphic mapping.

Proof. This is a local question over X, so we may assume that X is Stein and contractible. We follow the proof of Proposition 2 "with parameters" . The restriction of P to each of the open sets X x D and X x (E \ {x}) is holomorphically trivial. The transition co cycle comparing these two trivi­alizations is a holomorphic mapping X x D* -+ Gte; this gives a mapping X -+ G(O(D*», and the induced mapping X -+ r\G(O(D*» is holomor­phic. The composite mapping X -+ r\Q /P is equal to 1/1, which is therefore holomorphic. •

Lemma 1 implies in particular that the natural inclusion B(E, Gc)<--+

Unitary Representations of the Teichmiiller Group 29

Bund(E, Ge) is holomorphic. In order to introduce holomorphic line bundles on r\g /P, we recall

the central extension 9 -+ 9 by C* defined as in [P-S). Let r -+ rand p -+ P be the induced central extensions. Then we have the crucial

Lemma 2. (Segal reciprocity law) [Seg) [K-N-R) The central extensions r --; rand P -+ P are canonically split, so that r~r x C* and P~P x C·.

Using this lemma, we may view r and P as subgroups of 9. Then we have the C*-bundle r\9/p -+ r\9/p.

Lemma 3. The C*-bundle r\9/p -+ r\9/p is holomorphic (i.e., the transition cocycles are holomorphic functions.

Thus we have a holomorphic line bundle Lo -+ r\9/p = Bund(E, Ge). The restriction of Lo to B(E, Ge) C Bund(E, Ge) will be denoted by Lo too.

We have

Proposition 3. [D-N) We have Pic(B(E, Ge) = Z, and the class of Lo is a generator of Pic(B(E, Ge».

Therefore we can write the holomorphic line bundle LV associated to a representation V of Ge as L~mv. The integer mv is described very precisely in [K-N-R); if ch(V) = E>. n A • eA is the character of V, then mv = ! . n>. . (-X, 8)2, where 8 is the coroot corresponding to the highest root O. There is another equivalent description of mv in topological terms: we have a commutative diagram

H3(G,Z)

1= Z mv

---+

H3(SL(V), Z)

1= Z

The equivalence of these two descriptions follows from [Dy), and was known to Kumar.

The interesting phenomenon is that the g.c.d. of the integers mv, as V runs over all finite-dimensional representations of Ge, is not always equal to 1 (these g.c.d.'s are tabulated in [K-N-R]). This is in complete analogy with the fact that in general, the Steinberg-Matsumoto central extension by K2(k) of a split simple simply-connected group G(k) over a field k is not always induced from that of SL(n, k) by a representation of G; indeed the integers mv are the same in both cases.

We are now in a position to state the main theorem of [B-L) and of [K-N-R). We will use the characters X : r -+ C* and X' : P -+ C* corresponding to the splittings of the central extensions.

30 J.-L. Brylinski and D. McLaughlin

Theorem 1. [B-L] [K-N-R] The vector space HO{13{E, Gc), L~) identifies with the space of holomorphic functions f : 9 -+ C such that

f{hgh') = X{h)kX'{h')k . f{g) (2)

for any hEr and any h' in P. In other words, we have an isomorphism between HO{Bund(E, G), L~)

and HO(13{E, Gc), L~), given by restriction of holomorphic sections. We will also use the formal power series version of this Theorem. This

depends on the fact that Bund(E, G) identifies with the double coset space

G(C[[z]])\G(C{{z)))/"P. There is a central extension G(C{(z))) -+

G(C{(z))), and we have:

Theorem 1'. [B-Ll [K-N-R] The vector space HO{13(E, Gc), L~) identifies

with the space of holomorphic functions f : G(C{(z))) -+ C such that

f(hgh') = X(h)kX'{h')k . f(g) (2')

for any hEr and any h' in P.

Note that this formulation has the advantage that we have the freedom to choose any formal coordinate z at p (i.e., z can be taken to be any element of the completed local ring OE,p which vanishes at p to order exactly 1).

The vector space HO(13(E, Gc), Lg) is the vector space E(E,p, z, G, k) associated in the WZW (Wess-Zumino-Witten) conformal field theory to the Riemann surface E, the point p of E, the local coordinate z at p, the simple compact Lie group G and the integer k 2:: 1. Verlinde [V] gave a non rigorous derivation of a beautiful formula for the dimension of E(E, G, k). For G = 8L(2), the Verlinde formula was proved rigorously by Szenes [Sz] and by Bertram-Szenes [B-Szl, by Daskolopoulos and Wentworth [D­W2l, by Narasimhan and Ramadas [N-R] , by Thaddeus [Th], and by Zagier (unpublished). Tsuchiya, Ueno and Yamada [T-U-Y] prove the Verlinde formula for any G by showing that the proof of the formula can be reduced to computations of vector spaces associated to a curve of genus o with punctures. Similar results were obtained by Faltings [F2]. Finally, Jeffrey and Kirwan have obtained a direct geometric proof which works for any simple Lie group [J-K], and is based on their non-abelian localization theorem.

Fix the group G and the integer k. Then the vector space E(E,p, z, G, k) is the fiber at (E,p, z) of a holomorphic vector bundle C = Ck over the moduli space M(g,oo) of triples (E,p,z) consisting of a Riemann surface E of genus g, a point p of E and a formal coordinate z

Unitary Representations of the Teichmiiller Group 31

at p. We have a universal family of Riemann surfaces p: X --+ M(g, 00), where each fiber of p is equipped with a point and a formal coordinate at this point. To describe the vector bundle e, it is enough to give the space of holomorphic sections of e over some open subset U of M(g, 00): it is the space of all holomorphic functions f : U x G(C(z))) --+ C such that for any y E M (g, 00), the restriction of f to {y} x M (g, 00) satisfies the equation (2').

Note that, following [B-S] we can descend e to a vector bundle over the moduli space M(g, 1) of triples (E,p, A), where A is a non-zero element of r;E. There is a mapping M(g, 00) --+ M(g, 1), which sends (E,p, z) to (E,p, (dz)p). In fact the group A of automorphisms of the formal power series algebra C[[zll acts naturally on M(g, 00) on the right; the action is fixed-point free, and the quotient identifies with M(g, 1). The action of A on c[[zll extends naturally to an action on G(C[[z]]), and thus we have an ad ion of A on holomorphic functions satisfying condition (2'). It follows that e is A-equivariant, hence it descends to a holomorphic vector bundle over M(g, 1).

2. Sugawara construction and integrable connections

In this section we use the Sugawara construction to define a connection on the vector bundle e over M(g, 1). First we need to recall the infinitesimal action of the Lie algebra W = C(z)) . :fz of meromorphic formal vector fields on the moduli space M (g, 00). This action is due to a number of authors: Beilinson and Schechtmann [B-S], Kontsevich [Kon] , Alvarez­Gaume, Gomez, Moore and Vafa [AG-G-M-VJ, Kawamoto-Namikawa­Tsuchiya-Yamada [K-N-T-Y]. We follow the presentation of [K-N-T-Y]. For n an integer ~ 0, denote by M(g, n) the moduli space of triples (E,p, z), where z is an order n jet of a holomorphic function at p. Then M(g, n) is a finite-dimensional manifold, and M (g, 00) is the inverse limit of the M(g,n). To compute the tangent space to M(g,n) at (E,p,z), we observe that M(g,n) is the moduli space of pairs (E,i : Spec(C[[z]l/zn) ~ E), where i is an embedding of (non reduced) analytic spaces. The infinitesimal deformations of such pairs are classified by the first cohomology group of the sheaf (on E) of derivations which vanish on the subscheme Spec(C[[zlll zn). This is the sheaf T( -en + 1) . p) = T ® O( -en + 1) . p), where T is the holomorphic tangent bundle of E and O( - (n + 1) . p) is the sheaf of regular functions which vanish at p to order at least n. Thus we have

r M(g, n)(E,p,z) = Hl(E, T( -en + 1) . p)). (3)

32 J.-L. Brylinski and D. McLaughlin

We compute this cohomology group using the "open covering" consisting of E \ p and the "formal completion" Ep of E at pj this formal completion is defined as Spec of the completed local ring OE,p" This yields:

d d Hl(E, T( -en + 1)· p» = C((z»· d/zn+1. c[[zll· dz + Vect(E \ {p}) (4)

where Vect(E\ {p}) is the Lie algebra of holomorphic vector fields on E\ {p} which are meromorphic at p. Taking the inverse limit over n, and noting that the intersection of all the spaces zn+l . c[[zll· tz is trivial, we obtain

d T M(g,n)(E,p,z) = q(z»· d/Vect(E \p). (5)

Hence we have a natural map </>: q(z»· tz -> T M(g, 00) from q(z»· d~ to the tangent bundle of M (g, 00).

Theorem 2. </> is a Lie algebra homomorphism.

We now discuss a generalization of the Sugawara construction. The point of the Sugawara construction is to realize the Virasoro Lie algebra inside the degree 2 part of some completion of the enveloping algebra of a loop algebra. We will need a formal power series generalization. So we star~om the formal loop algebra 9 ® C((z» and its central extension

9 ® q(z». As usual we have:

9 ® q(z» = 9 ® q(z» EB C· c

(as vector spaces) and the bracket is

[(P(z), a· c), (Q(z), b· c)] = ([P(z), Q(z)J, ResQ B(P, dQ) . c) (6)

where B is the Killing form on g. For e E g and m E Z, let e(m) E

9 ® q(z» be the element e(m) = (e ® zm, 0). Let gem) be the span of the e(m) for e E g. Now let c be a complex number, the central chargej we will need to assume that c is not equal to -g, where g is the so-called dual Coxeter number. Let U be the quotient of the enveloping algebra U(g ® q(z))) by the ideal generated by the central element c - c· 1. Let U = Un Un be the standard filtration of U. We introduce on U the structure of a topological vector space for which a basis of neighborhoods of 0 consists of the left ideals 1m = U· gem) + U . gem + 1) + .... We give each Un the induced topology and let Un be the completion of Un. Then let U = Un Un. We note

Unitary Representations of the Teichmiiller Group 33

Lemma 4. 0 has a unique topological algebra structure which extends the algebra structure on U.

Proof. This follows easily from the fact that for U E g®C«(z», with order of pole at 0 equal to r, the map x ....... x· u sends each In to In- r . •

The point of the algebra 0 is that it acts naturally on any represen­tation V of 9 ® C((z» such that every x E V is killed by In for some n.

Now our purpose is to construct a Lie algebra homomorphism S : W = c«(z» . tz -+ U. The main property will be that S(v) implements the natural Lie algebra action of Won U, in the sense that

[S(v), x] = v· x for all x E U. (7)

We start with the known Sugawara construction (see [K-RJ) which gives an explicit expression of S( v) when v belongs to the Lie algebra iC[z, z-l] . tz of algebraic vector fields on C*. Let (ej) be a basis of g, and let (ej ) be the dual basis (with respect to B). The Sugawara construction is given by L:j [ej, [e j , xlJ = 2g . x for any x E g. Then we have:

n+1 d 1 ~ . , S(z . d) = 2(c + g) . ~ : ej(k). d(n - k) :E U

kE7.

(8)

The "Wick ordered product" : ej(k). ej(n - k) : is equal to ej(k)· ej (n - k) if k :5 n- k, and to ej(n- k) ·ej(k) if k > n - k. We note the commutation relation:

[S(zn+l . ~), S(zP+1 . ~)] =(p - n) . S(zn+p+l . ~) ~ ~ ~ (9)

n3 -n c + Dn+ o· -- . dim(g) . -- . 1

P. 12 9 + c

To define S( v) for arbitrary v, we write v = L:n~p anZn+ 1 1z, and we

put S(v) = L:n anS(zn+1 tz); the sum is convergent because S(zn+l d~) belongs to I[~l' By continuity we see that equation (7) is verified by S(v). The uniqueness of S(v) with this property is guaranteed by the following

Lemma 5. There is no non scalar element of 0 which commutes with

9 ® C«(z».

Proof. It is enough to show that for n ::::: 1 there i~o element of Un/Un- 1

which commutes with the adjoint action of 9 ® C«(z». Now the space

34 J.-L. Brylinski and D. McLaughlin

Un/Un- 1 identifies with a space of vector-valued formal Laurent series in n variable~ith values in g®n. An element in Un/Un- 1 which is invariant

under g ® C((z» must also be invariant under all the Sugawara operators S(v) E U, since any such operator is the limit of a sequence of elements in the enveloping algebra of g ® c((z». Since the action of the group of diffeomorphisms of the circle on the circle is n-transitive, the action of W on formal Laurent series in n variables is generically infinitesimally transitive, which implies that the only W-invariant Laurent series are constant. So any W-element invariant element in Un/Un- 1 belongs to g®n. However no such element can be invariant under g ® C((z», as a well-known argument shows. •

The S(v) obey the following commutation relation

(10)

where w is the 2-cocycle on W corresponding to the central charge dim(g)· g~c. This is a known fact for the usual Sugawara construction, and by continuity it holds true for any two elements of W.

We will apply the Sugawara construction to the central charge c = k. The representation of dim(g) . _c_ which will be relevant to us is a

g+c space V of holomorphic functions f on G(C((z»). To define V, we need the subgroup r(n) of r consisting of elements which are congruent to the identity modulo zn. Then V is the space of those holomorphic functions f on G(C((z))) which satisfy

(l)f(gh) = X(h)k . f(g) for all 9 E G(C((z»), hE 15 (2) for some n, we have: f(hg) = f(g) for all 9 E G(C((z))) , hE r(n)

This space V carries a representation of g ® C((z», which yields a structure of the U-module. Therefore the operators S(v) make sense on V, and satisfy the commutation relations (10).

We will need the following property of the Sugawara construction with respect to a point x = (E,p, z) of M(g, 00). Let j:l = j:lx C W be the Lie algebra of formal vector fields which are holomorphic on E \ {p}.

Lemma 6. Let w E V be a vector which is annihilated by the Lie algebra j:l. Then for any v E g®O(E\ {p}), the Sugawara operator S(v) annihilates w.

We are now ready to describe the connection on the vector bundle [; over M(g, 00).

Unitary Representations of the Teichmiiller Group 35

Theorem 3. (1) There exists a unique holomorphic connection D on £k ----+ M(g,oo) such that for any open subset U of M(g, 00), for any section f: U x G(C«(z») ----+ C of £ over U and for any element v ofW, we have:

Dq,(v)f = 4>(v) . f + S(v) . f

where 4>( v)· f represents the action on f of the vector field 4>( v) on M (g, 00), and S( v)· f is the action on f of the left-invariant differential operator S( v) on G(C«(z))).

(2) The connection D is projectively integrable, that is to say, its cur­vature K is a scalar-valued 2-form on M(g,oo). We have:

(3) The connection D descends to a connection on the vector bundle £ over M(g, 1).

Proof. First we need to verify that the expression 4>(v) . f + S(v) . f does define a section of £. This means we should ~ify that for every point of M (g, 00 ), the corresponding function on G (C( (z ) » satisfies a left­invariance and a right-invariance condition. The left-invariance condition is immediate because the first term involves a vector field on M(g, 00) and the second term involves S(v), which acts as an element of the completed enveloping algebra of g ® C«(z», acting as left-invariant differential oper­ators on G(C«(z))). As for the right-invariance condition, it is enough to show that 4>(v) . f + S(v) . f is annihilated by the action of e(x), where x f--+ e(x) is a holomorphic mapping from U to g ® C«(z», such that e(x) belongs to Px for any x E Uj here e(x) acts as a left-invariant vector field on G(C«(z))). We then have:

e· (4)(v), f + S(v)· 1) = 4>(v)e· f + S(v)e· f - [S(v),e]· f = -[S(v),e]· f

By the commutation relation (7), this is equal to -(v·e)· fj here v·e denotes the action of v E Won e E g ® C«(z», where v is viewed as a derivation of g ® C((z». Now since the derivation v preserves the Lie subalgebra Px, and since f is itself annihilated by Px, we see that -(v· e) . f = O. So we have shown that the operator Dq,(v) maps a germ of section of £ to a germ of section of £.

There is then one more thing we need to show to ensure that the formula for Dq,(v) does indeed define a connection. We must show that if we have vectors Vn of Wand holomorphic functions gn over an open subset U of M(g, 00) such that the vector field E gn4>( vn ) is identically zero,

36 J.-L. Brylinski and D. McLaughlin

then we have I: gn· D",(vn )! = 0 for any! : U x G(C«z») -+ C satisfying equation (2'). The equation I: gn¢(vn) = 0 is equivalent to the condition that for any x = (E, p, z) E U, I: gn (p )vn belongs to the Lie subalgebra j:Jx of G(C«z))). Then our claim follows from Lemma 6. Thus we have proved (1).

The curvature K of the connection D is as always given by K(~, "') = D1e.1I] - [De,DlI ). Take ~ = ¢(vt) and", = ¢(V2) for vectors VI,V2 E W. Then we have:

and on the other hand

Subtracting these two equalities and using the commutation relation (9) we obtain

proving (2). Now (3) follows from the easy fact that D is equivariant with respect to the A-action, and that M(g, 00) -+ M(g, 1) is a principal A-bundle. •

We note the explicit nature of the holomorphic connection in Theorem 3, as well as its analogy with the "heat equation connection" on classical theta-functions [Brl) [Be-Ka) [Hi). We note that there are many differ­ent constructions of this connection, due to G. Segal [Se), to Hitchin [Hi), to Axelrod-Della Pietra-Witten [A-DP-W), Tsuchiya-Yamada-Ueno [T­Y-V), Beilinson-Kazhdan [Be-Ka), Faltings [Fl) and Beilinson-Ginsburg [B-G). These connections probably all coincide up to the addition of some scalar-valued I-forms, but as far as we know this has not been verified. The construction of Hitchin for a connection on the vector bundle £ in­volves a quadratic term which is similar to the Sugawara term S( v) in Theorem 3. The method used in this section to construct the projectively integrable connection does not lead to a connection over Mg = M(g,O) it­self so easily. It is interesting that the curvature (which is scalar-valued) is a holomorphic 2-form, in contrast with the curvature computation done by Axelrod-Della Pietra-Witten [A-DP-W) which showed that the curvature is of type (1,1). This does not lead to any topological contradiction, since the cohomology class of type (1,1) over M(g, 0) pulls back to the trivial class in H2(M(g, 1), q.

Unitary Representations of the Teichmiiller Group 37

3. Hermitian holomorphic line bundles

There are many natural examples of holomorphic line bundles which come equipped with a natural hermitian structure; for instance, the Knudsen­Mumford line bundle is equipped with the Quillen metric [Qj. In some cases there are two different ways of defining two a priori different metrics, and it is possible to prove that the curvature of the two metrics coincide. Thus Deligne [De2] was able to conclude that the Quillen metric can be constructed by methods of algebraic geometry, in the case of line bundles over Riemann surfaces.

This suggests a systematic investigation of holomorphic line bundles with a hermitian metric. Let then X be a complex manifold. We define the notion of hermitian holomorphic line bundle to consist of a pair (L, h), where L is a holomorphic line bundle over X and h : L -t R+ is a hermitian form on L (so that h(z . p) = Izl2 . h(p) for z E C, pEL). There is a natural notion of isomorphism between two hermitian holomorphic line bundles. The group of isomorphism classes of hermitian holomorphic line bundles will be denoted by PiCh.h.(X). We wish to describe this group geometrically. Note that a hermitian holomorphic line bundle is the same as a Coo line bundle C equipped with two structures:

(1) a holomorphic structure, that is to say the notion oflocal holomor­phic invertible sections of C, which form a torsor under the sheaf Ox;

(2) a hermitian structure, that is to say a reduction of the structure group to 'll' c C*.

These two structures can be encoded into a tech l-cocycle with coef­ficients in the two-term complex of sheaves

(11)

in which Ox E9 Ix is placed in degree o. Here for any Lie group H, H x denotes the sheaf of smooth H -valued functions on X.

Let Ui be a good open covering of X consisting of Stein open sets. Let Si be an invertible holomorphic section of Cover Ui and let ti be a smooth invertible section of Cover Ui such that h(ti) = 1. Let gij (resp. Uij) be the transition cocycles with respect to the Si (resp. the ti); so Sj = gij . Si

over Uij and tj = Uij . ti. Then gij is a l-cocycle with coefficients in the sheaf Ox, and Uij is a tech l-cocycle with coefficients in the sheaf Ix. If we define a C* - valued function Pi on Ui such that ti = Pi . Si, then we have: Pj . pi! = gijui/· It follows that the triple (gi/ E9 Uij, Pi) defines a tech l-cocycle with coefficients in the complex of sheaves Ox E9 Ix -+ ~;.. By this process we obtain

38 J.-L. Brylinski and D. McLaughlin

Proposition 4. The cohomology class of (gij EB Uij,Pi) in H 1(X,Ox EB Ix -+ ~x) is independent of all the choices. In this manner the group Pich.h.(X) of isomorphism classes of hermitian holomorphic line bundles over X is identified with Hl(X, Ox EB Ix -+ ~x).

Note that given the holomorphic sections Si, we can choose ti = h(si)-! . Si, in which case we have Pi = h(si)-! and Uij = Igijl'

gij Proposition 4 should be viewed as an analog of a fundamental result of

Kostant [Ko], found by him in the mid 60's, which identifies the group of isomorphism classes of smooth line bundles over a Coo manifold X e~uipped with a connection with the hypercohomology group Hl(X,~x~Ai-), where Ai- is the sheaf of smooth I-forms with complex coefficients. This group is called a smooth Deligne cohomology group in [Br2] [Br-ML]. As demonstrated by Kostant, it is important to consider a line bundle together with a connection as a basic object; in fact, the Kostant central extension of a group of symplectomorphisms, first constructed in [Ko], consists of isomorphisms between such objects.

In the holomorphic context, there is a similar result, due to Deligne [Del] (see also [M-M]). First recall that for a complex manifold X, the Picard group Pic(X) is the group of isomorphism classes of holomorphic line bundles over X. This Picard group identifies with the cohomology group Hl(X, 0xJ. Now we have the exponential exact sequence of sheaves

0-+ Z(I) -+ Ox~Ox -+ 0,

hence Hl(X, Ox) identifies with the hypercohomology group H2(X, Z(I) -+

Ox). The complex of sheaves Z(I) -+ Ox is the Deligne complex of sheaves Z(I)v of order 1. Therefore we obtain Pic(X) = H2(X,Z(I)v).

Now Deligne [Del] identifies the group of isomorphism classes of holo­morphic line bundles over X equipped with a holomorphic connection, with

1 d log 1 1 • f the hypercohomology group H (X, Ox -Ox), where OX IS the sheaf 0

holomorphic I-forms. Using the exponential exact sequence again, we see that the complex of sheaves Ox ~Oi- is quasi-isomorphic to the complex of sheaves

Using multiplication by 21TH, we can replace the first term Z(I) by Z(2). Then we get the Deligne complex of sheaves Z(2)D, with a shift of degree by 1. Therefore the hypercohomology group Hl(X, Ox ~Ok) identifies with the Deligne cohomology group H2(X,Z(2)v). Deligne cohomology

Unitary Representations of the Teichmiiller Group 39

groups have many nice properties, including the existence of cup-products. They have been used by Deligne [Del], Beilinson [Be] and Bloch [BI] to construct holomorphic line bundles with connection by cohomological methods.

In view of the analogy of Proposition 4 with these results of Kostant and of Deligne, it is natural to denote by Z(l) D h.h. the complex of sheaves, with Ox EB I~ placed in degree 0, and to call it the hermitian holomor­phic Deligne complex of order 1. The hypercohomology of the complex of sheaves Z(l)D h.h. will be called the (order 1) hermitian holomorphic Deligne cohomology groups of X.

This viewpoint will be further developed later in this paper. We will need a variant of the groups studied by Kostant and Deligne: let us con­sider pairs (L, D) where L is a holomorphic line bundle over the complex manifold X and D is a connection which is compatible with the holomor­phic structure; this means that for any germ s of holomorphic section of L, the L-valued 1-form Ds is purely of type (1,0). Then we have

Proposition 5. The group of isomorphism classes of pairs (L, D) con­sisting of a holomorphic line bundle and a connection compatible with the holomorphic structure, identifies canonically with the hypercohomology group Hl(X, Ox ~A~'O»), where A~'O) is the sheaf of 1-forms of type (1,0).

Proof. We will describe the isomorphism for later use. Let (Ui ) be an open covering of X such that there exists an invertible holomorphic section Si of Lover U i . Let (gij) be the transition functions, so that Sj = gijSi over Uij.

Let Wi be the 1-form over Ui such that DSi = Wi®Si; we have Wi E A(l,O)(Ui )

since D is compatible with the holomorphic structures. Then (gij, -Wi)

is a 1-coc&cle of the covering with coefficients in the complex of sheaves Ox ~ Ai ,0) . The rest of the proof is similar to the proof of Kostant's theorem. •

Now it is a well-known result of A. Weil and Griffiths [Gr] that given a holomorphic line bundle L with a hermitian structure, there exists a unique connection on L which is compatible with both the holomorphic and the hermitian structures. This connection will be called the canonical connection. This gives, at the level of isomorphism classes of objects, a homomorphism Pich.h.(X) -. Hl(X,Ox~A~'O»). It is very interesting for us to construct this homomorphism cohomologically. For this purpose, we replace the complex of sheaves Ox EBIx -. ~~ by the quasi-isomorphic

40 J.-L. Brylinski and D. McLaughlin

double complex

A~'O) EI1 A . Ak,R -+ Ak i d log(9d log i d log (12)

Ox EI1 Ix -+ ~x

where Ak,R is the sheaf of real-valued I-forms.

Lemma 7. The double complex (12) is quasi-isomorphic to Ox EI1 Ix -+

~x' Proof. Since we have: Ak = A·Ak,REI1A~,o), the top row of the double complex is acyclic, hence the double complex (12) is quasi-isomorphic to its bottom row. •

Lemma 7 implies that Hl(X, Ox EI1 Ix -+ ~x) identifies with the first hypercohomology group of the double complex (12). It will be useful to write down a Cech l-cocycle with coefficients in (12) corresponding to the Cech l-cocycle (g:;/ EI1 Uij, Pi) with coefficients in Ox EI1 Ix -+ ~x' The bottom row components of this l-cocycle will be again gi/ EI1uij and Pi. We need to add one new component in the top row of (12), namely a Cech 0-cochain with coefficients in A~'O) EI1A'Ak,Ri there is only one possibility, namely -2d' 10g(Pi) EI1 [d' 10g(Pi) - d" 10g(Pi)]. Thus gi/ EI1 Uij, Pi and -2d' 10g(Pi) EI1 [d' 10g(Pi) - d" 10g(Pi)] are the three components of a Cech l-cocycle.

We then have a morphism ~ of complexes of sheaves from the double complex of sheaves (12) to the complex of sheaves Ox ~ A~'O), which we think of as a double complex standing in the O-th column. ~ is of course zero on the right column, and on the left column it is given by the horizontal arrows in the diagram

Then we have

A(I,O) EI1 . q. Al -x v-~ -X,R i d log (9d log

0* X

Id(90 --+

Id --+

Proposition 6. The map PiCh.h.(X) -+ HI(X, Ox ~A~'O)), which to a holomorphic hermitian line bundle associates the corresponding line bundle with connection, is the map induced on cohomology by the morphism of complexes ~.

We show how this proposition gives the computation of the curva­ture of a hermitian holomorphic line bundle (with respect to its canonical

Unitary Representations of the Teichmiiller Group 41

connection). First the curvature K of a holomorphic line bundle with ad­missible connection is the opposite of the 2-form obtained from its class in

1 ( d log (10». P'\ d logA(l 0) H X, Ox ----tAx' by the morphlsm of complexes from Vx ----t -x' to

0-> A~ which is given by d: Ll~'O) -> Ll~. This is clear because K = dwi

on Ui in our notations. Thus K can at most have components of type (1,1) and (2,0).

Then it follows that for a hermitian holomorphic line bundle, the cur­vature is equal to K = 2d(d' 10g(Pi» = 2d"d' log(pi). If we choose ti = h(Si)-! 'Si, so that Pi = h(Si)-!' we find that K = -d'd" log h(Si) =

d"d' log h(Si) where Si is any invertible holomorphic function on Ui' Of course K is purely of type (1, 1), and is purely imaginary. This is the curvature computation which can be found in Weil's book [WeI], except for the difference in our notations. We note the following cohomological interpretation of the curvature:

Lemma 8. The curvature of a hermitian holomorphic line bundle is ob­tained from its cohomological class in the first hypercohomology of the dou­ble complex of sheaves (12) by using the morphism of double complexes of sheaves from (12) to

A(l,l) -x

T o

which is given by d" : A~'O) -> A~,I).

-> o

T -> o

Now we will write down the main exact sequence which describes the group PiCh.h.(X). Let r2(X) denote the group of closed differential forms f3 on X of type (1,1) such that the cohomology class of (21Tv=1)-1 . f3 is integral. Then clearly the curvature of any hermitian holomorphic line bundle belongs to r2(X). Indeed the integrality of the cohomology class of (21TyCI)-l . K is a well-known result of Weil [WeI] and Kostant [Ko]. We have

Proposition 7. The group Pich.h.(X) sits in an exact sequence

where the map PiCh.h.(X) -> r2(X) associates to a hermitian holomorphic line bundle its curvature.

Proof. This can of course be proved by a direct method. Instead we use our sheaf-theoretic description of Pich.h.(X). We have an exact sequence

42 J.-L. Brylinski and D. McLaughlin

of sheaves

where 1I' is the constant sheaf, and A~llR cl denotes the sheaf of real closed 2-forms of type (1,1). The map ¢> i; defined by ¢>(f) = CSd'd" log (f) , where CS denotes the imaginary part of a 2-form. This means that the hypercohomology sheaves of Ke are 1f.o K e = 1I' and 1f.l K e = A~~R,cl. The hypercohomology spectral sequence then reduces to the exact sequence:

The map from HI (X, K e ) = Pich.h. (X) to A (1,1) (X, lR)cl associates to a hermitian holomorphic line bundle its curvature. Finally, the kernel of the map A(l,l)(X,lR)cl ~ H2(X, 1I') is equal to r2(X). •

The map H1(X, 1I') ~ PiCh.h.(X) has a simple interpretation: the group H1(X,1I') is the group of isomorphism classes of flat hermitian line bundles over Xj any flat unitary line bundle gives rise to a hermitian holo­morphic line bundle.

Proposition 7 is the analog for hermitian holomorphic line bundles of Kostant's description [KoJ of the group of isomorphism classes of pairs (L, D) of a line bundle and a connection on it.

From Proposition 7 we obtain a sort of Hartogs theorem for hermitian holomorphic line bundles.

Corollary. Let U be an open subset of the connected complex manifold X, such that Y = X \ U is a complex-analytic subset, and such that HI (U, 1I') is a torsion group. Then the restriction map Pich.h.(X) ~ Pich.h.(U) is injective and further for an element (L, h) of Pich.h. (X) the following two conditions are equivalent

(1) the curvature of (L, h) extends to a smooth 2-form K on X, such that the cohomology class of 2.".A is rational;

(2) some tensor power (L®n, hn) is in the range of the restriction map Pich.h.(X) ~ Pich.h.(U).

Proof. Since the map 71"1 (U) ~ 71"1 (X) is surjective, the restriction map H1(X, 1I') ~ H1(U,1I') is injective. Comparing the exact sequences of Proposition 4 for X and for Y we see that the restriction map is injec­tive. Next, it is clear that (2) implies (1). Conversely, if (L, h) satis­fies (1), there exists a hermitian holomorphic line bundle (L1' ht) over X such that the restriction of (L1.h1) to U has the same curvature as (L,h).

Unitary Representations of the Teichmiiller Group 43

Then (L®L1ix,hh11 ) has trivial curvature so it is given by an element of HI (U, '.0.') which by the assumption is killed by some integer n. Then (Llgm, hn) extends to a hermitian holomorphic line bundle over X. •

This corollary means that given a holomorphic hermitian line bundle (L, h) over U, some tensor power (L®n, hn) can be extended to a hermitian holomorphic line bundle over X if and only if the curvature form extends; furthermore, the extension to X is unique up to isomorphism. It is easy to give a geometric description of the extended line bundle L~n over X. For Van open subset of X, a holomorphic section of L~n over V is defined to be a holomorphic section s of L®n over V n U such that the function h(s)

over V n U extends to a smooth function on V. The hermitian metric h extends then to a hermitian metric for L~n in the obvious way. It follows from the above Corollary that this description is correct. If one wished to verify this directly, two non-obvious points would require proof: first, that sections s with h(s) bounded exist locally on Y, and second that Lx is indeed a holomorphic line bundle.

We will need to extend the description of hermitian holomorphic line bundles and Proposition 7 to a singular complex algebraic variety X. First we note that the notion of a holomorphic hermitian line bundle makes good sense in that more general context, and that Proposition 4 is still true. To proceed further, we need the notion of holomorphic differential forms on X and of smooth differential forms. Locally, when X is embedded in Cn ,

the differential graded algebra O·(X) of holomorphic differential forms on X is the quotient of O·(Cn ) by the differential graded ideal generated by the ideal I of X. Then there is no difficulty in patching this to define a sheaf Ox of differential graded algebras on X. These are the holomorphic differential forms in the sense of Grothendieck. Concretely, an element of OP(X) is a holomorphic section ofthe coherent sheaf AP (Jj J2) on X, where J cOx x x is the ideal of functions vanishing on the diagonal X C X xX.

Similarly we have the notion of smooth differential forms on X, de­noted by AP(X). We note that the de Rham cohomology, defined as the cohomology of this complex, is isomorphic to singular cohomology. Indeed, this follows by Weil's method (see [B-TJ), because there exist smooth par­titions of unity subordinated to any given open cover, and because any point on X has a neighborhood which admits a smooth contracting action of lR~o, so that the Poincare lemma holds true. In contrast, the Poincare lemma fails in general for holomorphic differential forms. The smooth dif­ferential forms have a decomposition into pure types (p, q), just as in the smooth case, and the exterior differential d splits as d = d' + d".

Next there is the notion of a connection for a line bundle L over X,

44 J.-L. Brylinski and D. McLaughlin

and this as always means that for any local section s there is an associ­ated section V's of Ak ® L, satisfying the Leibniz rule. Any smooth line bundle admits a connection by the usual partition of unity argument. The curvature is then a globally defined closed 2-form K, such that 21rA has integral periods. Conversely, any such differential form is the curvature of some connection on some line bundle.

The notion of a hermitian form on a line bundle is easily defined, and hence we have a group PiCh.h. (X). The structure of this group is then given by the following generalization of Proposition 7, in which r2(X) denotes the group of closed 2-forms on X of type (1,1) such that

(1) the cohomology class of 21rA is integral;

(2) K is locally of the form d'd" f for a function f. This second condition was automatically satisfied in the smooth case,

by the d'd"-lemma.

Proposition 7'. With r2(X) as above, the statement of Proposition 7 holds true for a complex algebraic variety X.

We will need the following sort of Hartogs theorem for singular alge­braic varieties.

Lemma 9. Let X be a complex algebraic variety, and let U be a Zariski open subset of X. Assume that Hl(X, 1I') and Hl(U, 1I') are finite groups. Then for an element (L, h) of Pich.h. (U), the following two conditions are equivalent:

(1) the curvature of (L, h) extends to a smooth 2-form K on X, which is locally in the image of d'd" and such that 21rA has rational cohomology class;

(2) some tensor power (L~m,hn) extends to an element of Pich.h.(X).

Proof. This follows immediately from Proposition 7' applied to PiCh.h. (X) and to Pich.h.(U). •

We note that we do not claim that the restriction map Pich.h. (X) ~ PiCh.h. (U) has finite kernel. To prove this, we would need to assume that there are no closed smooth 2-forms on X which have zero restriction to U.

We will later apply the Corollary to the case where U = 8(E, Grc) is the moduli space of stable Gc-bundles over a Riemann surface E, and X = 8(E, Gc) is the moduli space of semistable Gc-bundles.

Unitary Representations of the Teichmiiller Group 45

4. Holomorphic families of moduli spaces of stable bundles

In the rest of this paper we will discuss our second approach to the inte­grable on the vector bundle t:.

From now on, we will assume that G = SU(N), so that Gc = SL(N, C). We let S be the symmetric bilinear form

where the trace is taken in en. Given a closed oriented surface E of genus g, with base point p, there

is a moduli space B = B(G,g) consisting of conjugacy classes of homo­morphisms p : 7r 9 = 7rl (E, p) -t G which are such that the centralizer of the image of p reduces to Z(G) (such homomorphisms p will be called irreducible). Narasimhan and Seshadri [N-S] showed that B is a smooth manifold, and Atiyah and Bott [A-B] showed that B is in fact a symplectic manifold of dimension equal to (2 - 2g) . dim(G). It will be important for us to write down the symplectic form w on B. For this purpose, we recall from [We2] that the tangent space to B at the class of p identifies with Hl(7rg, g), where g is viewed as as a 7rg-module via p. We recall that there is a canonical orientation class 0 E H2(7rg,JR) = H2(E,JR). The 2-form w is then defined by

wee, T/) = (0, see, T/)} (13)

where see, T/) is an element of H2(7rg, JR). Here we use the cup-product on group cohomology. We note that the action of 7r on g gives a local system of Lie algebras 9 on E, and we have Hi (7rg , g) = Hl(E,g). The Killing form defines a unitary structure on the local system g, and then the symplectic pairing w on Hi (E, g) is given by cup-product.

The relation of this moduli space B with the moduli space of stable Gc-bundles over a Riemann surface E is given by a theorem due to Narasimhan and Seshadri [N-SJ for SU(n) and to Ramanathan [RJ in general. The theorem says that the natural map B -t B(E, Ge), mapping a flat G-bundle to the corresponding holomorphic Gc-bundle, is a diffeomorphism. There results a complex structure on M depending on the complex structure on E . This complex structure is given by Hodge theory for the cohomology group Hi (E, g), which identifies with the coherent sheaf cohomology group Hl(E, OE®g); this latter group has an obvious complex structure. We note that the theorem of [N-SJ and [RJ gives more information; in particular, any stable holomorphic Gc-bundle P -t E admits a unique reduction Q -t

E of the structure group to G c Ge, where Q -t E has the structure

46 J.-L. Brylinski and D. McLaughlin

of a flat G-bundle. The theorem of Griffiths (already used in §3 for line bundles) then tells us that there is a unique connection on P --+ ~ which is compatible both with the holomorphic structure and with the reduction of the structure group to G; the latter condition means that this connection is induced by a connection on the G-bundle Q --+ ~. This connection will again be called the canonical connection.

We introduced in §l the notion of Poincare bundle over an open set U C B(~, Gc); this is a holomorphic Gc-bundle P --+ U x~. For such a Poincare bundle, we have a smooth restriction Q --+ U x ~ of the structure group to G; indeed for each y E U, we have such a subbundle Qy C P/{y}XE,

and these subbundles organize into a G-bundle Q --+ U x~. Then by Grif­fiths's theorem there is a unique connection (the canonical connection) on the Poincare bundle which is compatible with the holomorphic structure and with the reduction of the structure group to G. Any isomorphism of Poincare bundles is automatically compatible with these canonical connec­tions.

We will take 71'g to be the group generated by (al,"" ag, bl,' .. ,bg) with the relation [al, bd· .. lag, bg) = 1. Let Tg be the genus 9 Teichmiiller space (see [Bers)), so that a point of Tg is an isomorphism class of pairs (~, 4», where ~ is a compact genus 9 Riemann surface and 4> is a class of isomorphisms 71'1(~)"'::'7I'g modulo inner automorphisms (this makes sense because 71'1 (~) is well-defined up to inner automorphisms, without choosing a base point). Recall that the Kodaira-Spencer map identifies the tangent space TE,,,, Tg with Hl(~, T), where T is the holomorphic tangent bun­dle. We have a universal family of curves 71' : X --+ Tg , together with isomorphisms 4>x : 71'1(71'-1 (x»"'::'7I'g up to inner automorphisms, depending smoothly on x = (~, 4» E Tg such that the element (7I'-1(x),4>x) of Tg is equal to x.

It will be important for us to construct a holomorphic fibration la --+ Tg with fiber over (~, 4» E Tg equal to B(~, Gc). This is included in a result of Pandharipande [Pal. We will however briefly outline the construction of la, based on an extension of the Kodaira-Spencer theory to deformations of complex manifolds P with some extra structures; we assume the structures are such that there is no non-trivial automorphism of P equipped with them. This leads to an appropriate local universal deformation space with tangent space equal to Hi (P, e), where e is the sheaf of holomorphic vector fields preserving the given structures. In §2 we encountered the case where the extra structure on a Riemann surface is a base point together with an n-th order jet of coordinate at p. Here our complex manifold P will be the total space of a stable Gc-bundle T : P --+ ~, where ~ is a Riemann surface of genus g. There are two extra structures here:

Unitary Representations of the Teichmiiller Group 47

(1) the right Gc-action J-L : P x Gc ~ P, which of course determines ~ = P/Gc and the Gc-fibration.

(2) an isomorphism 7l"1 (~)~7l" 9 up to inner automorphisms.

By the extended Kodaira-Spencer theory, there will be a local moduli space :!aloe, and a holomorphic mapping :!aloe ~ Tg , which maps (P, J-L, ¢) to (~ = P / Gc, ¢). The sheaf 8 consists of those holomorphic vector fields on P which are infinitesimally Gc-invariant, i.e., commute with the infinites­imal action of 9 ® C. To compute the tangent space to the local moduli space :!aloe at (P,J-L) we note that 8 = T- 1 [T.TplGc, the inverse image to P of the sheaf [T.TpjGc over ~, which is the Gc-invariant part of the direct image sheaf T. Tp . We emphasize that the pull-back T-1 is in the sense of sheaves of groups, not in the sense of coherent sheaves.

In particular, the sheaf T- 1 [T.TplGc is constant along the fibers of T. Since the fiber Gc of T is simply-connected, we see that Hi (P, e) = Hl(~,T.8) = Hl(~, [T.TplGC). Now the vector bundle (or locally free sheaf) [T.TplGc is in the middle of the exact sequence

(14)

We now assume that 9 ~ 2, so that HO(~, TE) = O. Then we have a short exact sequence

So given a point (~, P, ¢) of :!aloe, we have an exact sequence

There is no obstruction in this deformation problem, because the coho­mology group H2(p, 8) = H2(~, (P.Th plGC) = 0 for dimension reasons. There should be no difficulty in the convergence arguments necessary to construct actual deformations of complex structures [Kodl due to the non­compactness of P, because by Gc-invariance one can use L2 norms over ~ itself.

This shows that the map :!aloe ~ Tg is submersive, and its fiber embeds into B(~, Gc) as an open set.

Therefore from :!aloe we get a complex structure on an open subset of B x Tg • These complex structures agree on the intersection of two open sets, hence they define a complex structure on B x Tg • The complex man­ifold we have constructed will be called :!aj we have a smooth holomorphic submersion q : :!a ~ Tg , and the fiber q-l(~, ¢) identifies with B(~, Gc).

48 J.-L. Brylinski and D. McLaughlin

At this point we again introduce the notion of a Poincare bundle over an open subset U of JR. For this we need the fiber product y = X X Tg JRj a point in Y is the isomorphism class of a quadruple (E, ¢>, p, P ---. E), where (E, ¢» is a point of Tg , p is a point of the Riemann surface E, and P ---. E is a stable Gc-bundle. Let Pl : Y ---. X and 112 : Y ---. JR be the two projections. Then for U some open set in JR, a Poincare bundle over U will be a holomorphic Gc-bundle over p;l(U), such that for each point y E U, the corresponding holomorphic Gc-bundle over the Riemann surface p;l (y) is the one parametrized by the point y E JR.

Just as in §1 we have that Poincare bundles exist locally, any two of them are locally isomorphic, and the sheaf Aut(P) of automorphisms of a Poincare bundle P is canonically isomorphic to the constant sheaf Z(Gc). Thus we have a gerbe over JR consisting of Poincare bundlesj the band of this gerbe is Z(Gc).

We will need a compactification of the morphism JR ---. Tg • There are two natural (singular) compactifications of the moduli space B(E, Gc) of stable Gc-bundles. The first one is the moduli space B(E, Gc) of semistable bundles. The second one, which we will also have to use, is the space E of conjugacy classes of homomorphisms 'Trg ---. G. E as a smooth manifold with singularities only depends on the topological surface E, but its com­plex structure depends on the complex structure of E. Narasimhan and Seshadri [N-S] proved that B(E, Gc) is a normal projective algebraic vari­ety. There is a holomorphic mapping B(E, Gc) ---. E, extending the identity mapping of Bj in other words, the compactification B(E, Gc) dominates the compactification E.

We have a complex-analytic space lB and a holomorphic mapping ij : lB ---. Tg , whose fiber at (E, ¢» is equal to E, equipped with the com­plex structure corresponding to E. Similarly, according to [Pal, there is a complex-analytic space iii and a holomorphic mapping q : iii ---. Tg , whose fiber at (E, ¢» is equal to B(E, Gc)j there is a morphism iii ---. lB.

We will need the following results concerning the codimension of the boundary of E, which are easy to verify:

Lemma 10. The codimension of B(E, Gc) \B(E, Gc) in B(E, Gc) is ~ 3, except in the following cases:

(1) G = SU(2) and g :::; 4; (2) G = SU(3) and g :::; 2.

For any representation Gc ---. Aut(V), we constructed in §1 the cor­responding holomorphic determinant line bundle LV over B(E, Gc) ~ B. We wish to extend this line bundle to the compactification B(E, Gc). We have the following information concerning the line bundle LV: its Chern class is represented by the 2-form mv . w, where w is as in (13).

Unitary Representations of the Teichmiiller Group 49

Lemma 11. The symplectic 2-form w of (13) on B extends to a closed smooth 2-form on the singular complex variety B.

Proof. The tangent space to B at the fiat bundle P -+ E identifies with H1(E,g), where 9 is the corresponding fiat Lie algebra bundle, which is unitary. We can define a skew-symmetric pairing on this tangent space in the same way as we defined w on the tangent space to B. Thus we obtain a 2-form on B which restricts to won the open set B. To see that this 2-form is closed, we note that its pull-back to the space of all fiat connections on E is closed, by a computation due to Hitchin. •

Lemma 11 implies that, up to possible torsion, the first Chern class of Cv extends to a cohomology class in H2(E,Z(1». To extend holomorphi­cally some tensor power of the holomorphic line bundle itself, we will work with B(E, Ge), rather than E, since B(E, Ge) is a normal variety. We note the following lemma observed by Hitchin in [Hi]

Lemma 12. Assume that B(E, Ge) \ B(E, Gc) has complex codimension 2:: 3 in B(E, Ge). Then the restriction map Hl(B(E, Ge), 0B(E,Gc» -+

Hl(B,OB(E,Gc» is bijective.

The condition of Lemma 12 is verified in all cases except in the excep­tional cases listed in Lemma 10.

Proposition 8. Assume that B(E, Gc)\B(E, Ge) has complex codimension 2:: 3 in B(E, Ge). Assume also that the 2-form K on E given by Lemma 11 satisfies the two conditions listed before Proposition 7'. Then we have

(1) There exists some integer n 2:: 1 such that the n-th power of the holomorphic determinant line bundle on B(E, Ge) has a unique extension to a holomorphic line bundle over B(E, Ge). This extension will also be denoted by c~n.

(2) For any extension of c~n to a holomorphic line bundle on B(E, Ge), the restriction map r(B(E, Ge), c~n) -+ r(B(E, Ge), c~n) is bijective.

Proof. We have proved all, except for the last statement, which follows from Hartogs' theorem applied to the normal analytic space B(E, Ge). •

5. Hermitian holomorphic degree 4 characteristic classes

The cohomological description in §3 of the group PiCh.h.(X) of isomor­phism classes of hermitian holomorphic line bundles may be viewed as the construction of an "enriched first Chern class" from PiCh.h. (X) to the

50 J.-L. Brylinski and D. McLaughlin

hypercohomology group HI (X, Ox ffi Ix --- ~x). We will need a simi­lar construction for degree four characteristic classes. We will start with the geometric data of a holomorphic Gc-bundle P ___ X over the complex manifold X, and of a reduction Q --- X of the structure group to G c Gc (it is not assumed that Q ___ X is a flat bundle). We have first the usual topological characteristic class", E H4(X, Z(2», corresponding to the gen­erator of the cyclic group H4(BGC, Z(2». This topological characteristic class has two refinements which are the secondary characteristic classes of Beilinson for one, and of Chern-Cheeger-Simons for the other:

(1) The holomorphic Gc-bundle admits a Beilinson characteristic class "'B in the Deligne cohomology group H4(X, Z(2)v), which we describe here as the hypercohomology group H3(X, Ox --- Ok). We note that Beilinson [Be] constructs this class for algebraic Gc-bundles, with values in Beilinson­Deligne cohomology (which is a refined version of Deligne cohomology, incorporating control of growth at infinity). Rationally, this class is given an explicit tech construction in [Br-ML2].

(2) The smooth G-bundle Q --- X, equipped with its canonical connec­tion, admits a Cheeger-Chern-Simons class "'ccs in the hypercohomology group H4(X, Z(2)'D), where Z(2)'D is the complex of sheaves

In fact, this class "'ccs is constructed in [Chern-S] and [Cheeger-S] as a differential character; later Esnault [E] observed that the group of differential characters is isomorphic to H 4(X,Z(2)'D). An explicit tech cocycle with values in Z(2)'D was constructed in [Br-ML3]. Note that Z(2)'D is quasi-isomorphic to the complex of sheaves

d log r-:; I d . r-:; 2 d r-:; 3 Ix ---+Y -1· AX,R---+Y -1· Ax,IR---+Y -1· AX,IR'

with Ix placed in degree 1. The quasi-isomorphism involves dividing by 21TH. SO we can think of "'ccs as living in H3(X,Ix~H' AklR~H·A~IR~A·A~R)' The class "'ccs maps to the topologi-

, , , d Jog 1 d cal class", under the natural map from H3 (X, Ix ---+ A . Ax ,IR ---+ A . A~,IR~A' A~,R) to H4(X, Z(I». One can read the Chern-Weil4-form o representing'" from the secondary class "'ccs: one applies to "'ccs the morphism of complexes of sheaves

Ix d log

A'Ai-,R d

A'A~,R d

A'A~,IR ---+ ---+ ---+

1 1 1 1 (27r.;::T) -1·d

0 --- 0 --- 0 --- A4 -X,IR

Unitary Representations of the Teichmiiller Group 51

We wish to compare the two secondary classes "'8 and "'ccs. The comparison necessitates considering a complex of sheaves to which both Ox -+ ni- and Z(2)'D map. A natural such complex of sheaves is

(16)

where F2 A~ = Ef)p~2 A~,m-p) is the second step in the Hodge filtration of the de Rham complex. We indeed have a morphism ¢> of complexes of sheaves, given by the vertical arrows which are inclusion maps:

Ox 1

C* -x

d log ----+

d log ----+

This is a morphism of complexes because n~ c F2 A~. Then we have the natural morphism '¢ of complexes of sheaves

J!:x d log

A'AtlR d

A'A~,lR d A·Al,lR ----+ ----+ ----+

'¢: 1 1 1 1 C*

d log Al A~/F2A~ Al/F2Al -x ----+ -x -+ -+

It follows from the results of [Br-ML2] [Br-ML3] that the classes "'8

and "'ccs have the same image in H3(X,~x~Ai- -+ A~/F2A~). Note that here we have performed a truncation of the complex of sheaves (16), because it is in the hypercohomology of the truncated complex that we know that "'8 and "'ccs have the same image.

In fact, a stronger result follows from [Br-ML2], namely if we let "'8 and "'ccs also denote the specific tech 3-cocycles representing the secondary classes, as precisely constructed in [Br-ML2J, then there exists a Cech 2-cochain 0: with coefficients in the complex of sheaves ~x ~ Ai- -+

A~ / F2 A~, such that

(17)

Then the pair ("'8 Ef) "'ccs, 0:) defines a tech 3-cocycle with coefficients in the double complex of sheaves

C* d log Al A~/F2A~ -x ----+ -x -+

Z(2)v h.h. TeanEllean TeanEllean Tean

Ox Ef) J!:x -+ ni- Ef) A . Ai-,R -+ A'A~,lR

52 J.-L. Brylinski and D. McLaughlin

where can is used to denote canonical morphisms of sheaves. The double complex Z(2)D h.h. is called the hermitian holomorphic Deligne complex of sheaves of order 2. The bottom row of this double complex of sheaves is put in vertical degree -1. The hypercohomology groups of Z(2)D h.h.

are called the hermitian holomorphic Deligne cohomology groups. There is a natural morphism of complexes from Z(2)D h.h. to Z(2)D and to the truncation T:51Z(2)D' of the complex Z(2)D' in degrees ~ 1 (which means getting rid of the last sheaf A . A~.IR)' Denote by T:51 "'ccs the image of "'ccs in the hypercohomology of T:51Z(2)D'.

We then have

Theorem 4. Let P -+ X be a holomorphic Ge-bundle equipped with a smooth reduction of its structure group to G c Ge . Then there is a canon­ical cohomology class "'D h.h. in H4(X, Z(2)D h.h,) which maps to the class "'B and to the truncated class T:51 "'ccs under the natural maps of hyper­cohomology groups.

In order to be able to recover "'ccs (not just its truncation T:51"'CCS)

from the class "'D h.h .• it is useful to observe that Z(2)D h.h is quasi­isomorphic to the double complex of sheaves

C* -x ~ Al -x -+ A~/F2A~ -+ A~/F2A~ lean lean lean lean

Ox EI11x -+ 01 EI1 A· Al,R -+ A·A~.1R -+ A·A~.IR·

(18) The last column of this double complex is indeed acyclic, so it is quasi­

isomorphic to the double complex obtained by chopping off that column, that is to Z(2)D h.h' Then we can complete Theorem 4 by the following statement:

"'D h.h maps to "'ccs under the natural morphism of double complexes of sheaves from (18) to Z(2)D.

In particular, this gives a means of recovering the Chern-Weil differ­ential form 0 from the cohomology class "'D h.h .. Specifically, a tech 4-cocycle with coefficients in the double complex of sheaves (18) has a compo­nent which is a O-cochain Vi with coefficients in A· AtJR and component f3i which is a O-cochain with coefficients in A~ / F2 A~. The co cycle prop­erty implies that there exists a global 4-form whose restriction to Ui is dVi. We have the relation Vi ~ df3i mod F2 A~ j therefore 0 belongs to F2 A4(X), as is well-known.

We now introduce transgression maps in hermitian holomorphic Deligne cohomology, relative to a proper holomorphic fibration f : Y -+ Z whose fibers are connected Riemann surfaces. Recall [G-S) that we have

Unitary Representations of the Teichmiiller Group 53

transgression maps H4(Y,Z(2» _ H2(Z,Z(1» (in ordinary cohomology) and H 4(Y,Z(2)D') _ H 2(Z,Z(1)D') (in smooth Deligne cohomology groups, a. k. as groups of differential characters). There is also a transgression ho­momorphism H 4(Y,Z(2)D) - H 2(Z,Z(1)D) in Deligne cohomology. These transgression maps are compatible, so that we have some commutative di­agrams. For hermitian holomorphic Deligne cohomology groups we have

Proposition 9. There is a natural transgression map fy ..... z H 4(Y,Z(2)D h.h.) - H2(Z,Z(1)D h.h.), which is compatible with the trans­gression maps in smooth Deligne cohomology and in holomorphic Deligne cohomology. In particular we have a commutative diagram

fy~z --Corollary. Let f : Y - Z be a proper holomorphic fibration with con­nected fibers of dimension 1. Let P - Y be a holomorphic Gc-bundle equipped with a reduction to G of the structure group of the correspond­ing Coo bundle. Then there exists a natural holomorphic hermitian line bundle (L, h) over Z whose curvature is the 2-form (21TH)-1 fy ..... z n, where n E A4(X) is the Chern- Weil representative of the characteristic class", E H4(Y,Z(2)).

Conjecturally, the holomorphic hermitian line bundle (L, h) is closely related to the Knudsen-Mumford-Quillen line bundle. At this point, we can only establish a relation at the level of the curvatures of these line bundles. To state this, let p : G - U(V) be a unitary representation of G in a finite-dimensional Hilbert space V, and let Gc - Aut(V) be the extension to Gc of this representation. Then the corresponding holomor­phic bundle V - Y comes equipped with a hermitian metric. Then we have the holomorphic determinant line bundle V - Z of Knudsen and Mumford [K-M) , as recalled in §1. The line bundle V is equipped with the Quillen metric [Q), whose construction involves the knowledge of the Ray-Singer analytic torsion for the antiholomorphic laplacian operator for the restriction of L to the fibers of f. In the following result we use the integer mv introduced in §1. We note that the Chern-Weil representative of the second Chern class C2(V) is equal to mv . n. The relation of the holomorphic hermitian line bundle of the Corollary to Proposition 9 with the Quillen line bundle is clarified by the following statement, which in­volves the virtual representation V - dim (V) . 1 of dimension 0, where 1

54 J.-L. Brylinski and D. McLaughlin

denotes the trivial one-dimensional representation. In this statement, it is necessary to choose some hermitian metric on the fibers of the holomorphic mapping f.

Proposition 10. The curvature of the determinant line bundle .cv -dim(v)ol equipped with its Quillen metric is equal to mv times the curvature of the hermitian holomorphic line bundle (L, h) of the Corollary to Proposition 9.

Proof. This follows immediately from the Riemann-Roch type formula for the curvature of the Quillen metric proved by Bismut, Gillet and Soule [B-G-S]. The curvature of the hermitian holomorphic line bundle .cv is the degree 2 component of the differential form

h chcV)' Td(7j )

in which Td(7j) denotes the Todd class of the relative holomorphic tangent bundle 71, equipped with the given hermitian metric. We have: Td(7j) = 1 _ Ue_u ' with U = Cl (71)· Since Cl cV) = 0, we have:

chcV) = dim(V) - C2cV) = dim(V) - mv . n.

It follows that we have:

Cl (.cv ) = -my . h n + :2 . dim(V) . h Cl (71)·

Subtracting dim(V) times the similar expression for the trivial representa­tion we obtain the claim. •

We would like to suggest that the line bundle (L, h) constructed in this article by cohomological means gives the correct normalization of the Quillen metric.

Finally we note that it is possible to give a geometric construction of the line bundle (L, h), using the 2-gerbe on Y introduced in [Br-ML1] [Br-ML2J.

6. Curvature computations

We will apply the constructions of the last section to a Poincare bundle over an open subset U of R Recall that such a Poincare bundle is a holo­morphic Gc-bundle P over the open subset p:;l(U) of Y = X XT9 B. There

Unitary Representations of the Teichmuller Group 55

is a reduction of the structure group of P to G c Gc . We have the proper holomorphic fibration P2 : p;l(U) - U. Thus we are in the situation of the Corollary to Proposition 9, and we obtain a hermitian holomorphic line bundle (L, h)p over U. It is important to note that (using the theory of 2-gerbes) we can in fact construct an actual line bundle, not just its cohomology class. The hermitian holomorphic line bundle (L, h)p depends on the chosen Poincare bundle P; two Poincare bundles are locally isomor­phic, hence so are the corresponding hermitian holomorphic line bundles; however the isomorphism between them is only unique up to some element of Z( Gc). Such an element of Z( Gc) induces an automorphism of finite or­der ofthe hermitian holomorphic line bundle (L, h)p. Therefore there is an obstruction (of finite order) to constructing a global hermitian holomorphic line bundle. Let m be the order of Z(Gc); then it is clear that the m-th power of the hermitian holomorphic line bundles (L, h)p glue together to a global hermitian holomorphic line bundle.

This line of reasoning, combined with the computation before Propo­sition 1, shows that for a unitary representation p : G - U(V) there is a global hermitian holomorphic line bundle (LV, h), which is locally iso­morphic to the determinant line bundle for the bundle V associated to any Poincare bundle, with the metric constructed in the last section.

We now wish to compute the curvature of the hermitian holomorphic line bundle (L, h) and to write it down as a multiple of w. More pre­cisely, as a smooth manifold, ]a is just the product B x Tg , and we will show that the curvature of LV is a multiple of w, viewed as a 2-form on the first factor. For this we will use the description of B as the quotient A/lat.irr./g of the space A//at.irr. of flat irreducible connections on the trivial G-bundle E x G - E, by the natural action of the gauge group g = Map(E,G). The space AI/at.irr. is a smooth submanifold of the affine space of connections, identified with g-valued I-forms on E. We will interpret Tg as the space of conjugacy classes of injective homomor­phisms 'IjJ : 'Trg - PSL(2,lR), whose image is discrete and cocompact. Let Homd. cc.('Trg,PSL(2,lR» be the manifold of such group homomorphisms, so that Tg = Homd. cc.('Trg,PSL(2,lR»/PSL(2,lR). The tangent space to Homd. cc.('Trg, PSL(2,lR» at 'IjJ is the space of l-cocycles 'Trg - sl(2,lR), and the tangent space to Tg is the cohomology group H 1 ('Trg,sl(2,lR». To ev­ery homomorphism 'IjJ E Homd. cc.('Trg,PSL(2,lR» there is associated the Riemann surface E = H/'IjJ('Trg), where H is the upper half plane. Then A is the space of'Trg-invariant elements in the space AH of connections over H.

We note that we have a complex structure on AH induced by the isomorphism p : AH = Al(H) 0 g"':"A(O,l) (H) 0 9 which associates to a

56 J.-L. Brylinski and D. McLaughlin

real I-form its component of type (0,1). It is natural here to use anti­holomorphic I-forms, as we will see later. We recall that the g-action on AH extends to a holomorphic action of the "complexified gauge group" ge = Map(H,Gc). This comes, via the isomorphism p, from the natural gauge action of ge on A(O,I)(H): 9 E ge, acting on the right, transforms A E A(O,I)(H) into g-1 . A . 9 + g-1 . d" g.

We now consider the tautological connection d + A on the trivial G­bundle over Homd.cc.('Trg,PSL(2,JR.» x AH x H, where A is the I-form in the H-direction parametrized by A E AH. This connection is invariant under the conjugacy action of PSL(2, JR.) on the first factor, so that we obtain a connection on the trivial Ge-bundle over Tg x AH X H. This connection is compatible with the reduction to G of the structure group. We now want to construct a holomorphic structure on the trivial Ge-bundle over Tg x AH X H, such that for an open subset U of Tg x AH X H, a smooth function 9 : U --+ Ge is holomorphic if and only the g-valued 1-form dg . g-1 + A on U is purely of type (1,0). For this, according to a theorem of Griffiths [Gr] it will suffice to prove the following

Lemma 13. The curvature of the connection A on the trivial Ge-bundle over Tg x AH X H is purely of type (1,1).

Proof. The curvature K of the connection A, restricted to ([1/J] , A) x H, is necessarily of type (1,1). Now we have

K(B,t;)""A,p = (B,t;)p

where 1/J E Tg, A E Al(H) ® g, p E H, B E Al(H) ® g is a tangent vector to AH flat at A, and t; is a tangent vector to H at p. If B is a holomorphic tangent vector, then it is antiholomorphic as a I-form on Hi therefore if t; is a holomorphic tangent vector, we have K(B,t;)""A,p = (B,t;)p = O. Similarly this quantity vanishes if Band t; are both holomorphic. Since the 2-form K vanishes on any other types of pairs of tangent vectors, the Lemma follows. •

Then from Griffiths' theorem we obtain

Lemma 14. There is a unique holomorphic structure on the trivial Ge­bundle over Tg x AH X H which is compatible with the connection d + A. Then this connection is the canonical connection with respect to the above holomorphic structure and to the trivial reduction of the structure group to G.

For a given 1/J : 'Trg --+ PSL(2,JR.), the space (AH)1I"g of'Trg-invariant connections identifies with the space of connections on the Riemann surface

Unitary Representations of the Teichmiiller Group 57

E. If (AH )::able is the open subset of (AH )11"9 consisting of connections for which the corresponding holomorphic Ge-bundle over E is stable, then the product manifold Tg x (AH )11"9 X H admits a natural holomorphic mapping to y. Indeed, the theorem of Narasimhan and Seshadri [N-S] implies that for fixed 'If; , the quotient (AH) ;:able / ge identifies with the quotient Af/at.irr./g. Then we have

Lemma 15. The holomorphic Ge-bundle over 11. x (AH )11"9 X H constructed above is locally isomorphic to the pull-back of any Poincare bundle; the isomorphism preserves the reductions of the structure groups to G C Ge.

This implies that the trivial Ge-bundle over Tg x AH X H, equipped with the tautological connection, can be used as a model for all curvature computations relative to a Poincare bundle. Our aim now is to compute the curvature of LV' We computed K in the course of proving Lemma 13. The expression for K will simplify if we restrict ourselves to AH flat.

Observe that the invariant bilinear form Son 9 corresponds to a generator of H3(G,Z).

Then from K and from the bilinear form S on 9 we obtain the Chern­Wei! 4-form n representing the characteristic class K,. We have

According to Proposition 10, the curvature of LV is equal to 211"A . mv . IE n. This gives the 2-form 1I":Er . W over Tg x AH flat.

Thus we have proved the following

Proposition 11. The curvature of the connection on the determinant line bundle LV on Tg x AH flat is equal to 1I":Er . w.

It is of course interesting to be able to extract an "n-th root" of this hermitian holomorphic line bundle, whenever the n-th root exists as a topo­logical line bundle. For a holomorphic line bundle L it is well-known and easy to see that any n-th root L~ of L as a smooth line bundle is automatic­ally a holomorphic line bundle. The same holds for hermitian holomorphic line bundles in view of the following

Lemma 16. The following diagram is cartesian

In other words, given an element (L, h) of Pich.h.(X) and a class "f E

H2(X, Z(1)) such that n"f = Cl (L), there exists a unique element (L', h') of

58 J.-L. Brylinski and D. McLaughlin

Pich.h.(X) whose n-th power is isomorphic to (L, h) and whose first Chern class is equal to 'Y.

Proof. We have an exact sequence of complexes of sheaves

o ~ [Z(l)EBZ(l) ~ Z(l)] ~ [OxEB1!h(1) ~ ~x] ~ [OiEBIx ~ ~x] ~ o.

The complex of sheaves Z(l) EB Z(l) ~ Z(l) is quasi-isomorphic to Z(l). We then have the exact sequence of hypercohomology groups

H1(X,Ox EBl!h(l) ~ ~x) ~PiCh.h.(X) ~ H2(X,Z(1)) ~

~ H2(X, Ox EBl!h(l) ~ ~x)

Since the extreme terms of this exact sequence are vector spaces over C, the result follows. •

If some n-th root of a determinant line bundle LV is defined globally on lB, then its curvature is equal to n.:J=r . w.

7. Speculations about the integrable connection

In genus 1 it is proved in [A-DP-W] that the projectively integrable con­nection over the vector bundle Ek over Teichmiiller space Tl admits a hori­zontal positive-definite hermitian form. This hermitian form is diagonalized by the basis of Ek consisting of Kac-Moody characters of level k [A-DP­W].

We would like to use the curvature computations of §6 to establish a similar property for the integrable connection in higher genus. These computations should be combined with the algebro-geometric construction of the connection due to Hitchin [Hi]. However there are some serious difficulties in doing this. At this point we will simply explain the hermitian inner product on Ek , under the assumptions that G = SU(N) and that g are such that the codimension of B(E, Gc) \ B(E, Gc) in B(E, Gc) is at least 3 (cf. Lemma 10 and Proposition 8). Here we choose k such that the k-th tensor power of the fundamental line bundle can be defined globally on lB. This line bundle over lB will be denoted by L®k. According to §5 and §6, we have on L®k the structure of a holomorphic hermitian line bundle. The curvature of the canonical connection D on L®k is equal to 'irA· w, according to the computations in §6.

The holomorphic vector bundle E ~ Tg has fiber at (E, 'I/J) equal to the space HO(B(E, Gc), L~) considered in §l (actually, in §l we considered the similar bundle over M(g, 00)). We now define a coherent sheaf E over Tg

Unitary Representations of the Teichmiiller Group 59

as equal to the direct image q .. L®k, where q : B -t Tg is the projection. In fact it is known that £ is in fact a vector bundle; for instance, this follows from the existence of a projectively integrable connection (see §2 and the references given there).

We now wish to construct a hermitian form on the vector bundle £. The definition is very simple

(21)

where 2N is the dimension of the fiber of q. To justify this definition, we need to show that

Lemma 17. Under the assumptions of Proposition 8, let S1 and S2 be holo­morphic sections of L~ over B(E, Ge). Then the function h(S1' S2) extends to a continuous function on the compactification B(E, Gc) of B(E, Gc).

Proof. It is enough to prove this for S1 = S2 = s. According to Proposition 8, there exists some tensor power L®km which extends to a holomorphic hermitian line bundle over B. If we show that h( s, s)m extends to a con­tinuous function on B, the same will hold true for h(s,s). Therefore we may as well assume that L®k extends to a holomorphic line bundle over B(E, Gc). By Proposition 8, s extends to a holomorphic section of over B(E, Ge). Since h is a continuous hermitian metric over B(E, Gc), h(s, s) extends to a continuous function on B(E, Ge). •

Now the 2-form w extends to a smooth 2-form on X by Lemma 10 and the fact that X dominates E. Therefore the integral (21) converges and defines a hermitian inner product on the vector bundle £.

It is natural to conjecture that the projectively integrable connection on £ preserves the hermitian form, and evidence for this is contained in [A-DP-W].

The idea behind the construction of the integrable connection should be that a tangent vector v on Tg would act by Dv + Pv, where Dv refers to the horizontal lift of vB::: Tg x B, and Pv is some differential operator 2 in the B-direction. According to [Hi], the symbol of Pv should be equal to the holomorphic section of S2(TB(E, Gc)) which corresponds to v. Hitchin proves the integrability of the connection, so the remaining difficulty is to rewrite his connection in a way that would make it clear that the hermitian form is horizontal. The sought-after expression should involve the connec­tion D, so that the properties of D could be used to establish that the hermitian form is horizontal.

60 J.-L. Brylinski and D. McLau.ghlin

Assuming this construction has been done, we would then want to use the vector bundle with connection (£, V) over Tg to define a projective representation of the Teichmiiller group r g , following [A-DP-W]. Recall that rg is the group of outer automorphisms of the group 'Trg • There is a natural action of r 9 on Tg given by

T' (E,1/J) = (E, f1/J),

where T E rg and f is any automorphism of'Trg representing T. This is a holomorphic action, which lifts in an obvious way to the complex man­ifold JR, and the symplectic form w is r g-invariant. Now we consider the holomorphic hermitian line bundle Lk over the complex manifold B, and note that its curvature is r g-invariant. We have: 'Trl (JR) = 'Trl (B), a finite group. Therefore, if we assume that k is divisible by a large enough integer, Kostant's construction of a central extension of a group of symplectomor­phisms [Ko] would apply, with small modifications, and we would obtain a central extension r 9 -+ r 9 of r 9 by '][' such that the holomorphic hermitian line bundle L®k is equivariant under r g. It then would follow that the vec­tor bundle £ -+ Tg is r g-equivariant, and that the action of r 9 preserves the connection and the hermitian inner product on £ .

Then the space r jlat(Tg , £) of flat global sections of £ would become a unitary representation of rg , hence a projective representation of r g •

There is an alternate approach to the construction of the action of r 9

on the line bundle L®k. Namely instead of trying to make the line bundle equivariant, one looks at the gerbe of Poincare bundles over B and one observes that this gerbe is equivariant under r 9 (the notion of equivariant gerbe was developed in [Br2]). Then one tries to make a global object of the gerbe equivariant under r g; this approach has the advantage of showing that the central extension r 9 can be taken to have finite kernel.

In any case it would then follow that there exists some integer m such that if k is divisible by m, the space of flat global sections of the vector bundle £k -+ Tg carries a unitary projective representation of r g'

It has been conjectured by Kontsevich that these representations fac­tor through a finite group. There is an interesting arithmetic phenomenon which creates some difficulty in attempting to use the hermitian inner prod­uct to establish this finiteness. From the point of view of quantum groups, the representation can be defined over some ring of cyclotomic integers (see [Koh] for SU(2» and carries an invariant hermitian form defined over that ring. This hermitian form is positive definite; however, its conjugates under the action of the Galois group do not appear to be positive definite. This is related to the fact that the Jones polynomial is non-negative at the

2 .. 13 root of unity q = e m but not at the other roots of unity [J].

Unitary Representations of the Teichmiiller Group 61

References

[A-DP-W] S. Axelrod, P. Della Pietra, and E. Witten, Geometric quantiza­tion of Chern-Simons gauge theory, J. DiJJ. Geom. 33 (1991), 787-902.

[A-B) M. F. Atiyah and R. Bott, The Yang-Mills equations over Riemann surfaces, Phil. Trans. Roy. Soc. London A 308 (1982), 523-615.

[AG-G-M-V] L. Alvarez-Gaume, C. Gomez, G. Moore, and C. Vafa, Strings in the operator formalism, Nucl. Phys. B 303 (1988),455-521.

[Be] A. A. Beilinson, Higher regulators and values of L-functions, J. Soviet Math. 30 (1985),2036-2070.

[Be-Ka) A. A. Beilinson and D. Kazhdan, Flat projective connections, preprint (1991).

[Bl) S. Bloch, The dilogarithm and extensions of Lie algebras, Algebraic K­Theory, Proceedings Evanston, Lect. Notes in Math. vol. 854 (1981), 141-167.

[B-G] A. A. Beilinson and V. Ginsburg, Resolution of diagonals, homotopy algebras and moduli spaces, preprint (1993).

[Bers) L. Bers, Spaces of Degenerating Riemann surfaces, Discontinuous Groups and Riemann Surfaces, Princeton University Press, (1974).

[Br-Ko) R. K. Brylinski and B. Kostant, Minimal representations, geomet­ric quantization and unitarity, PNAS USA 91 (1994), 6026-6029.

[B-L) A. Beauville and Y. Laszlo, Conformal blocks and generalized theta functions, preprint (1993).

[B-S] A. Beilinson and V. Schekhtman, Determinant bundles and Virasoro algebras, Commun. Math. Phys. 118 (1988), 651-701.

[B-Sz] A. Bertram and A. Szenes, Hilbert polynomials of moduli spaces of rank 2 vector bundles II, Topology 32 (1993), 599-609.

[B-T) R. Bott and L .'Ill, Differential Forms in Algebraic Topology, Grad­uate Texts in Mathematics, Springer-Verlag (1982).

[Br1] J-L. Brylinski, Loop groups and non-commutative theta functions, preprint (1989).

[Br2] J-L. Brylinski, Loop Spaces, Characteristic Classes and Geometric Quantization, Progress in Math. vol. 107, Birkhiiuser Boston (1993).

[Br-ML1] J-L. Brylinski and D. McLaughlin, A geometric construction of the first Pontryagin class, in: Quantum Topology, Series on Knots and Everything vol. 3, World Scientific (1993), 209-220.

[Br-ML2) J-L. Brylinski and D. McLaughlin, The geometry of degree four characteristic classes and of line bundles on loop spaces I, preprint (1992), to appear in Duke Math. Jour.

[Br-ML3] J-L. Brylinski and D. McLaughlin, Cech cocycles for character­istic classes, preprint (1991).

62 J.-L. Brylinski and D. McLaughlin

[ Br-ML4] J-L. Brylinski and D. McLaughlin, The geometry of degree four characteristic classes and of line bundles on loop spaces II, in prepara­tion.

[Chee-S] J. Cheeger and J. Simons, Differential characters and geometric invariants, Lect. Notes in Math. vol. 1167 (1985), Springer-Verlag, 50-80.

[Cher-S] S. S. Chern and J. Simons, Characteristic forms and geometric invariants, Ann. Math. 99 (1974), 48-69.

[Del] P. Deligne, Le symbole modere, Publ. Math. IHES 73 (1991), 147-181.

[De2] P. Deligne, Le determinant de la cohomologie, Current Trends in Arithmetical Algebraic Geometry, Contemp. Math. 67 (1987), 93-178.

[D-N] J-M. Drezet and M. S. Narasimhan, Groupe de Picard des varietes de modules de fibres semi-stables sur les courbes algebriques, Invent. Math. 97 (1989), 53-94.

[D-WJ G. Daskolopoulos and R. Wentworth, Factorization of rank two theta functions II. Proof of the Verlinde formula, preprint (1993).

[Dy] E. B. Dynkin, Topological invariants of linear representations of the unitary group, Dokl. Akad. Nauk 85 (1952), 697-699, (in Russian).

[E] H. Esnault, Characteristic classes of flat bundles, Topology 27 (1988), 323-352.

[F1] G. Faltings, Stable G-bundles and projective connections, J. Alg. Geom. 2 (1993), 507-568.

[F2] G. Faltings, A proof of the Verlinde formula, J. Alg. Geom, (to ap­pear).

[Gi] J. Giraud, Cohomologie non-Abelienne, Ergeb. der Math. 64 (1971). [Go] R. Godement, Topologie Algebrique et TMorie des Faisceaux,

Hermann, 3rd edition (1964). [Gr] P. A. Griffiths, Periods of integrals on algebraic manifolds 1. Con­

struction and properties of the modular varieties, Amer. J. Math. 90 (1968), 568-626.

[G-S] H. Gillet and C. Soule, Arithmetic Chow groups and differential char­acters, Algebraic K-Theory: Connections with Geometry and Topology, Con temp. Math. (1989), 29-68.

[Hi] N. Hitchin, Flat connections and geometric quantization, Commun. Math. Phys. 131 (1990), 347-380.

[I] B. Iversen, Cohomology of Sheaves, Universitext, (1986). [J] V. Jones, Hecke algebra representations of braid groups and link poly­

nomials, Ann. Math. 126 (1987), 335-388. [J-K] L. Jeffrey and F. Kirwan, Localization for nonabelian group actions,

preprint (1992).

Unitary Representations of the Teichmiiller Group 63

[K-N-T-Y] N. Kawamoto, Y. Namikawa, A. Tsuchiya, and Y. Yamada, Geometric realization of conformal field theories on Riemann surfaces, Max-Planck- Institut fUr Mathematik, preprint 87-53 (1987).

[K-M] F. F. Knudsen and D. Mumford, The projectivity ofthe moduli space of stable curves I: Preliminaries on "det" and "div", Math. Scand. 39 (1976), 19-55.

[Kod] K. Kodaira, Complex Manifolds and Deformation of Complex Struc­tures, Grundl. der Math. Wiss. 283 , Springer-Verlag (1986).

[Koh] T. Kohno, Topological invariants for 3-manifolds using representa­tions of mapping class groups I, Topology 31 (1992), 203-230.

[Kon] M. I. Kontsevich, Virasoro algebra and the Teichmiiller space, Funct. anal. Appl. 21 (2) (1987), (1986) 156-157.

[Ko] B. Kostant, Quantization and unitary representations, in Lect. Notes in Modern Analysis and Applications III, Lect. Notes in Math. vol. 170 (1970), 87-208.

[K-N-R] S. Kumar, M. S. Narasimhan and A. Ramanathan, Infinite grass­mannians and moduli spaces of G-bundles, preprint (1993).

[M-M] B. Mazur and W. Messing, Universal Extensions and One­Dimensional Crystalline Cohomology, Lect. Notes in Math. vol. 370 (1974).

[N-R] M. S. Narasimhan and T. R. Ramadas, Factorisation of general theta functions I, Invent. Math. 114 (1993), 565-623.

[N-S] M. S. Narasimhan and C. Seshadri, Stable and unitary vector bundles on a compact Riemann surface, Ann. of Math. 82 (1965),540-564.

[Pal R. Pandharipande, A compactification over Mg of the universal family of slope-semistable vector bundles, preprint (1993).

[P-S] A. Pressley and G. Segal, Loop groups, Clarendon Press (1986). [Q] D. Quillen, Determinants of Cauchy-Riemann operators over a Riemann

surface, Funct. Anal. Appl. (1985), 31-34. [Ram] A. Ramanathan, Stable principal bundles on a compact Riemann

surface, Math. Ann. 213 (1975), 129-152. [R-T] N. Reshetikhin and V. G. Turaev, Invariants of 3-manifolds via link

polynomials and quantum groups, Inv. Math. 105 (1991), 547-597. [Seg] G. Segal, The definition of conformal field theory, preprint (1988). [Ser] J- P. Serre, Un theoreme de dualite, Comment. Math. Helv. 29

(1955), 9-26. [S] A. Szenes, Hilbert polynomials of moduli spaces of rank 2 vector bun­

dles, Topology 32 (1993), 587-597. [Th] M. Thaddeus, Stable pairs, linear systems and the Verlinde formula,

preprint (1992).

64 J.-L. Brylinski and D. McLaughlin

[T-U-Y] A. Tsuchiya, K. Ueno and Y. Yamada, Conformal field theory on universal family of stable curves with gauge symmetries, Adv. Studies in Pure Math. 19 (1989), 459-565.

[V] E. Verlinde, Fusion rules and modular transformation in 2d conformal field theory, Nucl. Phys. B 300 (1988), 360-376.

[WeI] A. Weil, Varietes KaehIeriennes, Hermann (1957). [We2] A. Weil, On the moduli of Riemann surfaces, Collected Works,

[1958b]. [Wi] E. Witten, Quantum field theory and the Jones polynomial, Commun.

Math. Phys. 121 (1989), 351-399.

J-L. Brylinski, Department of Mathematics, The Pennsylvania State University, University Park, PA 16802

D. McLaughlin, Department of Mathematics, Princeton University, Princeton, NJ 08544

Received April 4, 1994

Differential Operators on Conical Lagrangian Manifolds

Ranee Brylinski* and Bertram Kostant**

1. Introduction

The suggestion that there is an operator to be obtained (and consequently a spectrum to be obtained) from a function on a symplectic manifold orig­inated from the work of physicists, on atomic radiation spectra, in the early part of this century. It is without doubt a profound and extremely important mathematical question to determine under what circumstances such operators naturally exist and, if possible, to construct them. The progression from function to operator generally goes under the name of quantization.

The problem is readily broadened, and ambiguities lessened, when it becomes a question of quantizing not just one function, but instead a Lie al­gebra (under Poisson bracket) offunctions. The result is then to be a linear representation of the Lie algebra. Under quite optimal circumstances there is a prescription in place for doing this - geometric quantization. The machinery of geometric quantization involves integral symplectic forms, quantum line bundles, prequantization, half-forms and polarizations. For the case of a coadjoint orbit it has been successfully applied in those in­stances where the optimal conditions are satisfied. For the more subtle coadjoint orbits, when the these conditions are not satisfied, the question of what to do remains largely unsolved. A classic example that something can be done, even when the optimal conditions are not present, is the con­struction of the metaplectic representation. Here, however, it is the very special intervention of the Heisenberg group which points the (very limited) way. The orbit in question is the minimal non-trivial coadjoint orbit of the symplectic group.

In the papers [B-K2] and [B-K3] we have announced a solution to the quantization problem for a simple Lie algebra DR of functions on a sym­plectic manifold M where M, via the moment map, corresponds to the real points of the minimal non-trivial coadjoint orbit Omin of Ad D and where a certain and necessary (Vogan) half-form bundle exists. Here g is the com­plexification of gR. For technical reasons we have excluded only the cases

* Research supported in part by a Sloan Foundation fellowship ** Research supported in part by NSF Grant No. DMS-9307460

66 R. Brylinski and B. Kostant

where g is not simple or the case where the symmetric space correspond­ing to gill is Hermitian. A consequence is a construction of the minimal (and unitary for a corresponding group) representation of glR which, un­like derived functor constructions, explicitly exhibits the operators 7I'(x) for x E !JR. Indeed it exhibits them as pseudo-differential operators. The nature of the construction is such that analytic information, such as certain matrix entries, and the Hilbert space inner product, can be readily deter­mined. As one might be led to expect from geometric quantization, the vectors in a dense subspace of the module are represented, geometrically, as half-forms on a suitable lagrangian submanifold Y of Omin' The almost total uniformity of a successful quantization procedure for the case at hand, in our opinion, represents a limited but nonetheless very real breakthrough for the problem of quantization.

The present paper is the first of a series giving the proofs of the the­orems in [B-K2] and [B-K3]. If gR = tlR + PIR is a Cart an decomposition of glR and g = t + 13 is its complexification, the problem readily reduces to constructing the operators 7I'(x) when x E Pill on a t-finite module V of half-forms on Y. In fact for x E PIR one will have that 7I'(x) = Ix - Tx is a sum where Ix is a multiplication operator and Tx is its adjoint with respect to the glR-invariant positive definite Hermitian structure B on V. The determination of B is essentially equivalent to the determination of the family F = {Txlx E Pill}. On the other hand F will be shown to be a commutative set of pseudo-differential operators. The main result in the present paper is Theorem 3.10. Theorem 3.10 is a key step in the con­struction of F. It is also interesting in itself since it constructs differential operators by combining the linear action of t on 13 with a rather elegant use of polynomial denominators. In fact Theorem 3.10 constructs for each x E Pill a differential operator, D x , which after multiplication by the in­verse of a quadratic polynomial in the Euler operator, is equal to Tx. The paper also contains additional constructions which will be needed in our subsequent papers. In particular we sketch at the end of §5 how we will implement the generalized Capelli identity of Kostant and Sahi [K-S] in order to explicitly compute the operators Tx on half-forms.

2. The orbit Y of highest weight vectors

2.1. Let t be a complex semisimple Lie algebra and let K be a connected complex Lie group with Lie algebra t. In this section we establish some properties of the space of sections of a homogeneous line bundle on the orbit of highest weight vectors in a simple K -module.

2.2. Let ~ c t be a CSA (Cartan subalgebra) and let bet be a Borel

Differential Opemtors 67

subalgebra such that ~ C b and let m be the nilradical of b. The pair (~, b) determines a set A+ = A+(~, m) of positive roots and a corresponding set p++ C ~* of dominant integral weights. If 1/J E p++ then let V", be the irreducible (finite-dimensional) representation of t with highest weight 1/J. Define 1/J* E P++ to be the highest weight of the dual representation V,;. Let FI"') C F be the V",-isotypic component in at-module F. Let VI' C V be the JL-weight space if ~ acts on a vector space V and JL E ~.

Let T C K be the maximal torus corresponding to ~. We say that a character X of T is K -dominant if dX E P++. Then the dual character X* is defined by the condition dX* = (dX)*' We call X self-dual if X = X*.

2.3. Let u be a simple finite-dimensional K-module so that u ~ V"Y where , E P++. Let Q C K be the stabilizer of the highest weight space u"Y so that Q is a parabolic subgroup. Let X be the character of Q defined by the action of Q on u"Y so that dxlT =,. Let Z+ denote the set of non-negative integers.

2.4. Let "( E (u*)-"Y be a (non-zero) lowest weight vector. Then

Y = K."( c u* (2 - 1)

is the orbit of highest weight vectors. We assume from now on (without loss) that, is non-trivial on each simple component of t. We have

JP(Y) ~ KIQ (2- 2)

If X is an algebraic variety then let R(X) denote the algebra of regular functions on X. Let V(X) be the algebra of global algebraic differential operators on an algebraic variety X and let vm(x) be the subspace of differential operators of degree at most m where m E Z+ (see A.5).

Let E E V 1 (Y) be the Euler vector field on the cone Y. (The Euler vector field on a vector space V, or on any cone inside V, maps each linear function to itself.) Notice E is K-invariant. The flow of E on Y is the algebraic action of C* on Y by dilations and therefore the eigenvalues of E on R(Y) are integral and the p-eigenspaces Rp(Y), p E Z, give an algebra grading.

2.5. Proposition. All regular functions on Y extend to the closure Y C u* so that we have

R(Y) = R(Y) (2 - 3)

Consequently all the algebraic differential operators on Y extend to Y so

68 R. Brylinski and B. Kostant

that

V(Y) = V(Y) (2 - 4)

Then Rp(Y) = 0 if P < 0 and we have a K -invariant algebra grading

(2 - 5)

The space R1(Y) generates R(Y) as an algebra. Furthermore Rp(Y)is a simple K -module equivalent to Vp-y, P ;:::: 0, so that

(2 - 6)

as K -modules. In particular Ro (Y) = C·l is the space of constant functions on Y. The variety Y is normal and Y = Y U {O}.

Proof. The normality, equality of functions and (2-6) are well-known since Y is an orbit of highest weight vectors (see e.g. 2.6(i) or [V-P]) and then (2-5) and the subsequent assertions are immediate. (2-4) follows immediately using A.6 since Y and Yare quasi-affine varieties with the same algebra of ~~~~. . 2.6. Remarks (i) Kostant proved not only the normality, (2-3) and (2-6) but also that the defining ideal I c S ( u) of Y c u * is generated by homogeneous quadratic elements in S(u) - see [Garl. Thus we know generators and relations for the algebra R(Y). (ii) Suppose that X is any C*-stable K-orbit in a simple K-module V with corresponding grading R(X) = EBpEzRp(X). If we assume that the boundary of X in its closure X c V has codimension at least 2 then R(X) is integral over R(X); moreover R(X) is the integral closure of R(X). But clearly Rp(X) = 0 if p < 0 and so we find that Rp(X) = 0 if p < O. These facts follow from the arguments in the proofs of Propositions 1.2 and 1.4 in [B-Kll.

2.7. We have a K-linear isomorphism

V 1-+ fv (2 - 7)

defined by (2-1). When the context is clear we will write "v" for "fv". Let Z, z E U be, respectively, non-zero highest and lowest weight vectors.

2.8. We have

Y~ K/Q' (2 - 8)

Differential Operators 69

where Q' = K' is the isotropy group. Then Q' C Q and Q' is the kernel of the character of Q defined by X-l. The K-homogeneous line bundles over Yare of the form

C,. = K x Q' C.,. -+ Y (2 - 9)

where r is a character of Q', C.,. is the I-dimensional Q'-module corre­sponding to r and the projection in (2-9) is given by (a, t) 1-+ a· (t(). The projection commutes with the natural actions of K on C.,. and Y. Hence the space of (algebraic) sections r(Y, C.,.) is both a completely reducible K-module and a module over the K-algebra R(Y).

2.9. Lemma. (i) The K -module r(Y, C.,.) is multiplicity-free. The K -type decomposition has the "ladder" form

(2 - 10)

where v E p++ but v - I rt P++. Each K -type in (2-10) is equivalent to the space of sections of some K -homogeneous line bundle over F(Y).

(ii) If So E r(Y, c.,. )rvl is a non-zero highest weight vector then {zP So I p E Z+} is a complete set of linearly independent highest weight vectors in r(y, C.,.).

(iii) r(Y, C.,.) is a faithful R(Y)-module and is generated over R(Y) by

r(Y, c.,. )[v)'

Proof. If q is a character of Q let La = K x Q Ca -+ F(Y) be the K­homogeneous line bundle defined in an analogous way to the bundle (2-9). The natural projection Y -+ F(Y) is a C* -bundle and moreover it identifies with the C*-bundle associated to Lx-I.

Let Q = Qo ~ U be a Levi decomposition of Q where U is the unipotent radical of Q and Qo :::> T. Then Q' = Q~ ~ U is a Levi decomposition where

Q~ = Q&. Notice that characters of Q (respectively, Q') are equivalent in the obvious way to characters of Qo (respectively, Q~). Now Q~ is the kernel of a character of the reductive (connected) group Qo and so it follows easily that r extends to a character T of Qo and that all such extensions are of the form TXi where i E Z. We then find that

(2 - 11)

Then (i) follows from the Borel-Weil Theorem and in particular Vv ~ r(F(Y), L:;:-x") where p is the largest integer such that this space of sec­tions is non-zero. Now (ii) and (iii) follow easily. •

70 R. Brylinski and B. Kostant

2.10. Notice that in Lemma 2.9, the line bundle G.r determines the "min­imal K-type" Vv uniquely. We will - by slight abuse of terminology -also refer to r(Y,OT)[V) as the "minimal K-type" in r(y,OT). We do not assume that dT is non-trivial on each simple component of t; in particular T may be the trivial character of K so that then r(Y, OT) = R(Y).

2.11. The dilation C*-action on the algebra R(Y) induces a C*-action on the algebra V(Y) which clearly preserves the order filtration and corre­sponds to the derivation ad Eon V(Y).

Let V(Y, OT) be the algebra of differential operators on sections of the line bundle OT (see A.5). In A.13 we define the notion of an t;,-lift on OT where t;, is a vector field on Y.

2.12. Lemma. OT admits a K-invariant E-lift E' E V(Y,OT). Then E' is diagonalizable on r(Y, OT) with spectrum equal to {t, t + 1, ... } where t E C. The (p + t)-eigenspace of E', p E Z+, is equal to r(Y, OT hv+P'Y) so that the K -module decomposition

(2 - 12)

coincides with the decomposition into eigenspaces of E'. Furthermore E' is unique up to the addition of a scalar. Thus the operator adE' on V(Y, OT) is independent of the choice of E'. We will call this operator adE.

Proof. The Euler vector field on Y corresponds to the right action of Q / Q' on Y which then lifts to a right action of Q/Q' on OT; this follows easily using the extension T given in the proof of 2.9. Then upon differentiating we obtain a K-invariant E-lift E'. But then it follows using A.13 that the complete set of K-invariant E-lifts is {E' + cjc E C}.

Now since E' is K-invariant and the sum (2-10) is multiplicity-free, E' acts on each K-type in (2-12) by a scalar. The rest of the lemma now follows easily using 2.9(i)-(ii). •

2.13. Let:Fp C :F denote the p-eigenspace of ad E whenever :F c V(Y, OT) is an adE-stable subspace. Then 2.12 implies that we have a well-defined K-invariant algebra grading

(2 - 13)

and we have Rp(Y) c Vp(Y,OT)' p E Z+.

2.14. Remark. By 2.5, the vertex {OJ of the cone Y is exactly the singular locus and the complement of {OJ is the K-orbit Y. Thus, in view of A.ll

Differential Operators 71

and (2-3)-(2-4), the failure of surjectivity of the natural map

R(Y) ® U(t) -+ 1J(Y, CT ) (2 - 14)

when CT is trivial is in some sense a measure of the "badness" of the singularity at O. Indeed we prove in §3 that (2-14) is not surjective - see 3.17. We have determined explicitly 1J(Y) and the image of (2-14) - we will give this in a subsequent paper.

3. Differential operators on homogeneous line bundles over Y

3.1. From now on we assume that the parabolic subgroup Q is of a rather special type: Q is the parabolic subgroup associated to a Hermitian sym­metric pair (t, to) of tube type. Then, in this section, we construct an abelian subalgebra A_ C 1J(Y, CT ) of algebraic differential operators on Y which is "new" in the sense that A_ has only trivial intersection with the image of (2-14).

3.2. Let q be the Lie algebra of Q. From now on we assume that Q arises in the following way: there is a semisimple element h E t such that the eigenvalues of ad h on tare ±1, 0, the sum of the eigenspaces corresponding to -1 and 0 is equal to q and furthermore t admits an S-triple (2h, e, e) (so that [h, e] = e, [h, e] = -e and fe, e] = 2h). Let ~i C r be the i-eigenspace of ad h if ~ c t is an ad h-stable subspace. Then

(3- 1)

and

q = to + L1 (3- 2)

Then (3-2) is a Levi decomposition and L1 is the nilradical of q. Clearly then t1 and L1 are abelian. The space bo = ~ + rno is easily seen to be a Borel subalgebra of to.

Let K be the simply-connected covering group of K with covering map K -+ K. Let Ko C K and Ko C K be the (connected) subgroups corresponding to to. Then Q = Ko X K-1 is a Levi decomposition where K ±1 _C K are the unipotent subgroups corresponding to t±1' Let KG = (Ko)( so that KG is the kernel of x-1IKo ' Then Q' = KG ~ K-1 is a semidirect product so that a character of Q' is equivalent in the obvious way to a character of KG' Let q' be the Lie algebra of Q'. Then, q' = t( so

72 R. Brylinski and B. K ostant

that we have a direct sum

q = q' + Ch (3 - 3)

3.3. A pair (t, to) occurs in 3.2 if and only if (t, to) is a complex Hermitian symmetric pair of tube type; i.e., to is the fixed point algebra of a complex Lie algebra involution of t and the corresponding non-compact real sym­metric space has a K-invariant Hermitian structure and is of tube type. Furthermore the pair (t, to) is then the direct sum, in the obvious way, of irreducible complex Hermitian symmetric pairs of tube type (t(i) , t~i») where t(i) is a simple component of t.

Let q be the rank of the symmetric pair (t, to). The next result 3.4 is well-known (see e.g. [K-S]). Then the passage to the case 3.6 where t is semisimple follows easily by considering the decomposition of t into a direct sum of simple subalgebras. Notice that we may regard elements of 8(t1) as elements of 8(t) via the obvious inclusion 8(t1) C 8(2).

3.4. Theorem. Assume the Lie algebra t is simple. Then the polynomial algebra 8(td is a completely reducible multiplicity-free Ko-module. The ring of highest weight vectors

is a polynomial ring in q algebraically independent homogeneous generators P1, .•• Pq where Pj E 8j (t1 ). The polynomial Pq E 8(t) is a highest weight vector for the adjoint action of t on 8(t). Furthermore Pq is a Ko-semi­invariant of weight X5 where XO is a K -dominant character of Ko; moreover xo generates the character group of Ko. Finally XO is self-dual, i.e., the corresponding simple K -module V")'o' where "Yo = dXo, is self-dual.

3.5. Table 1 lists the pairs (t, to) occurring in 3.4 along with q, the dimen­sion of Y and the corresponding representation V")'o. This list is well-known - see e.g. [He, p. 528, Example 4] or [K-S, p. 89].

Table 1.

t to q dimY V")'o

sl(2n,q s(gl(n, q + gl(n, q) n n2 + 1 /\nC2n

so(n+2,q so(n, q + so(2, q 2 n+l cn+2

sp(2n,q gl(n,q n n(n+1) + 1 /\~C2n 2 so(4n,q gl(2n,Q n n(2n - 1) + 1 C!-spin

E7 E6 +so(2,Q 3 28 C56

Differential Operators 73

In Table 1, t\~C2n denotes the irreducible representation of sp(2n, C) satisfying t\~C2n + t\n-2C2n :::::1 t\nc2n. In the next line, C!-spin is a half­spin representation of so (4n, C) of dimension 22n- 1 • In the final line, C56 is the unique 56-dimensional irreducible representation of E7 •

3.6. Corollary. The polynomial algebra S(t1) is a completely reducible multiplicity-free Ko-module. The ring of highest weight vectors S(tt}'l1o is a polynomial ring in q algebraically independent homogeneous generators. If P E S(t1) is a Ko-semi-invariant then P E S(t) is a highest weight vector for the adjoint action of t on S(t). Jilurthermore the weight of P is X~) where Xp is a K -dominant character of Ko. Finally every K -dominant character of Ko is self-dual and arises in this way.

3.7. Now 3.6 implies that \) :::::1 V")' is self-dual. Notice that in the situation of 3.4, 'Y is necessarily of the form 'Y = no where p E Z+.

Let

p : U(t) ...... End \) (3 -4)

be the natural representation. We may write u·v for (pu)(v) where U E U(t) and v E p. Since tl is abelian we have a natural identification

(3- 5)

3.8. Lemma-Definition. There exists a unique Ko-semi-invariant poly­nomial P E S(tl) such that

(pP)(z) = z

Furthermore P has weight X2 (see 2.3).

Proof. It follows from 3.2 that

(3 - 6)

(3 - 7)

Consequently (3-6) holds for some P E S(tt}. But, since \) is self-dual, z and z are Ko-semi-invariants of weights X and X-1, respectively. The lemma now follows easily using the first statement in 3.6. •

3.9. Since E is K-invariant, the K-action on r(Y,CT ) (corresponding to the K -action on CT ) defines a Lie algebra homomorphism

(3- 8)

74 R. Brylinski and B. Kostant

where we recall the algebra grading (2-13). Here we are suppressing in the notation the dependence of (3-8) on r; we will write 1f.,. if needed. Then 1f(x), x E t , is an 1]x-lift on C.,. (see A.13) where 1]x is the vector field on Y corresponding to x. Notice that (2-7) carries p(x) to 1f(x) for each x E t.

Let {vm (Y, C.,.) }mEZ+ be the order filtration of V(Y, C.,.) (see A.5) and let {Ui (t) hEZ+ is the usual filtration on the universal enveloping algebra U(t). Then R(Y) = V°(y, C.,.) by (A-8). Furthermore easily we have

(3- 9)

By (3-5), P defines a differential operator 1fp E Vo(Y,C.,.) and so a map

1fp : r(Y, C.,.) -+ r(Y, C.,.) (3 - 10)

3.10. Theorem. The image of the differential opemtor (3-10) is equal to the R(Y)-submodule zr(Y, C.,.). Consequently the equation

(3 - 11)

where s E r(Y, C.,.) defines a (non-zero) algebmic differential opemtor of degree -1

(3 - 12)

The multiplication opemtor z on r(Y, C.,.) commutes with 1f P so that we may write Dz as the quotient

D _ 1fp z-

Z (3 - 13)

The K -tmnslates of Dz span a finite-dimensional simple K -submodule A-I C V-I (Y, C.,.) equivalent to tl, and thus the assignment z 1-+ Dz ex­tends uniquely to a K -linear isomorphism

(3 - 14)

The opemtors Dv , v E tl, commute and genemte a gmded abelian subal­gebm A_ = ffip;o:oA_p of V(Y, C.,.). Finally, we have a K -linear algebm isomorphism

(3 - 15)

where Df" = Dv if v E tl.

Differential Opemtors 75

Proof. Let r = r(Y, CT ) and let r j = r(Y, CT ) [v+i"rJ' j E Z+. Let TO be the character of Ko such that dTo = v. Then TO is a self-dual K-dominant character because of 2.9(i) and 3.6.

Let s E roY be a non-zero lowest weight vector for the t-action. Then zPs E r p , p E Z+, is a Ko-semi-invariant lowest weight vector for the t-action so that we find using 3.2 that

(3 - 16)

Let U E U(tt). Then

(7rP)(7rU)(szP) = (7ru)(7rP)(szP) (3 - 17)

since P and U commute as elements of U(tl). First we claim that (7rp)ro = O. Since P E S(t) is a highest weight vector for the K-action by 3.6, it suffices to show that ift = (7rP)(s) then t = o. Now using (3-16) and the complete reducibility of the Ko-action on S(tt} we find that if s E ro is a non-zero highest weight vector for the t-action then s = (7ra)(s) where a E S(tt)2v is a Ko-semi-invariant (cf. 3.8). Suppose that t i= O. Then it follows using 3.6 that a = Pb for some Ko-semi-invariant b E S(tl)2a where 0: E p++. But then 2v = 2')' + 20: and consequently v - ')' E P++. This contradicts 2.9(i) and proves the claim.

Next let p 2: 1. We claim that there is a scalar cp E C* such that

(3 - 18)

Indeed szP and zszp-l are non-zero Ko-semi-invariants in r p of weights TO -lX-p and TO -lX-(p-2) respectively. Using (3-16) and completely re­ducibility again, we find that zszp-l = (7rM)(szP) where M E U(tt) is a Ko-semi-invariant of weight X2. But then P = cpM where cp E C* since the Ko-action on U(tt) is multiplicity-free by 3.6. This proves (3-18) and in particular that 7r P i= O. Thus we find

(3 - 19)

because of (3-17), (3-18) and the fact

(3 - 20)

which is evident as z E U is a highest weight vector. We conclude that (7r p)r c zr. Moreover the reverse inclusion follows easily by essentially "reversing" the argument.

76 R. Brylinski and B. Kostant

Thus (7I"P)r = zr. Consequently, by A.7, (3-11) defines a differential operator D z E V -1 (Y, CT ). Moreover, it follows using A.6 that we can prove any identity in "D(Y, CT ), e.g., the commutativity of two operators, by showing the identity is true for the corresponding operators on reV, CT ).

Now the operators 7I"P and z commute because of (3-20). Further­more 7I"P E 7I"(U(t)) c Vo(Y,CT ) and z E R 1(Y) c V 1(Y,CT ) are high­est weight vectors for the K-action of weights 2')' (by 3.6 and 3.8) and ')' respectively. Since the K-action on V(Y, CT ) is locally finite (A.12), it follows from (3-13) that Dz E V-l(Y,CT ) is a highest weight vector of a K-submodule A_l ~ u so that we have (3-14). Let W C "D-2(Y, CT) be the subspace spanned by the commutators [Dv, Dv'l where v,v' E u. Then W is a K -submodule of V -2 (Y, CT ). Suppose Wo C W is a non-zero sim­ple K-submodule with Wo ~ VI" JJ E P++. Since W is the image of the commutator map 1\2 A-l -+ W, the K-type VI' occurs in 1\2 u. It follows that 2')' > JJ in the usual order on highest weights. But also, for some p 2:: 0, the natural map Wo -+ Hom(r v+2 , r v) is non-zero. So VI' occurs in Vv+(v+2h ® Vv+n' It follows that JJ 2:: 2')'. Contradiction. Thus W = O.

Finally let A_v C V-v(Y, CT ) be the image of the multiplication map SV(A_l) -+ A-v- We have the natural maps A_v -+ Hom(rHv, r j ),

j E Z+. Arguing as above, it follows that a K-type VI' occurs in A_v only if P'Y 2:: JJ and JJ 2:: P'Y. Since the K-type Vp-y occurs just once in SV(A_l) we conclude that either A_v = 0 or A_p ~ Vn . We rule out the former possibility since (3-18) implies that the pth power D! is non­zero. Thus A_ ~ R(Y) as K-modules by (2-6). But then it follows that the two algebra homomorphisms S(u) -+ A_ and S(u) -+ R(Y) (defined, respectively, by v 1--+ Dv and v 1--+ Iv for v E u) have the same kernel and hence we obtain the algebra isomorphism (3-15). •

3.11. Corollary (of proof). Let p E Z+. Then reY, CT ) [v+p")'l, and so in particular Rp(Y), is multiplicity-free as a Ko-module.

Proof. Immediate from (3-16) and 3.6. • 3.12. The simplest example of 3.10 occurs when t = s[(2, C), to is the subalgebra of diagonal matrices, V")' ~ ((;2 is the standard representation and T is trivial. Then Y = ((;2 - {(O, On so that R(Y) = qz, zl is a polynomial ring in the two functions z, z and z, z are coordinates on Y (see A.14). Then V(Y) = qz, z, tz, tzl is the Weyl algebra. We find immediately that 7I"P = ztz so that Dz = tz' Notice then that A_ qtz' tzl and R(Y) = qz,zl generate V(Y). See also 3.17.

Differential Opemtors 77

3.13. Now the action of K on Y defines a natural action of K on T*Y. In this way, T*Y, equipped with its canonical symplectic structure, becomes a Hamiltonian K-space. These structures are algebraic (cf. [B-Kl, p. 274]). The corresponding moment map

/1: T*Y -+ t* (3 - 21)

then identifies with the collapsing map K xQ' (t/q')* -+ t*. The algebra homomorphism defined by pullback of functions is

4> : S(t) -+ R(T*Y) (3 - 22)

On the other hand corresponding to the canonical projection

ty: T*Y -+ Y (3 - 23)

defining the cotangent polarization we have an algebra inclusion

R(Y) C R(T*Y)

3.14. Lemma. The restricted map

(3 - 24)

is injective. Furthermore (3-24) is "order preserving"i i.e., if U E sm(t1) (cf. (3-5» then 7rU has order mi moreover

E(7ru) = 4>(u) (3 - 25)

Proof. By (3-1)-(3-3), we have q' n t1 = O. Thus the natural map (t/q')* -+ ti is surjective. Consequently the morphism /11 : T*Y -+ t* -+ ti is surjective where /11 is obtained by composing the moment map (3-21) with the natural map t* -+ ti. But then the algebra homomorphism

4>1 : S(tt} -+ R(T*Y) (3 - 26)

corresponding to /11 is injective and 4>1 = 4>ls(tt}. Suppose U E sm(t1) with U =I- O. Then, we have 7rU E VO'(Y, Or) by

(3-5) and (3-9). But easily we find that the order m symbol of 7rU (see A.I0 and (A-20» is Em(7ru) = 4>(u). Since 4>(u) =I- 0 we conclude that the E(7ru) = 4>(u) and 7rU has order equal to m (see (A-27». It follows easily now that (3-24) is injective. •

78 R. Brylinski and B. Kostant

3.15. Define 1 by 2l = deg P so that

(3 - 27)

Then 3.14 and A.7 give the following corollary to 3.10.

3.16. Corollary. The differential operator D,. has order 2l and symbol

(3 - 28)

In particular the rational function ¢P / z on T*Y is in fact regular. Every nonzero differential operator DE A-l has order equal to 21.

3.17. Remark. We have A_p C 'D_p(Y, GT ), p E Z+. Thus, if we choose some E' as in 2.12 and consider the corresponding grading on r(Y, GT ),

then we find the elements of A_ act on r(Y, GT ) as lowering operators while the elements of R(Y) act as raising operators. In particular then if S is the image of (2-14) then A_ n S = 0 since S C EBp~o'Dp(Y, GT ) by (2-5) and (3-8). In a subsequent paper we will discuss the generation of the algebra 'D(Y, GT ) - see also 4.6.

4. The Hermitian pairing

4.1. Let

H = r(Y,GT ) (4 - 1)

Let tJR be a compact real form of t and let j : t -t t be the corresponding complex conjugation map defined by j(x + iy) = x - iy where x, y Eta. We may choose tJR so that hE ita. Let KR C K be the maximal compact subgroup corresponding to tIR.

The purpose of this section is to prove Theorem 4.5 below which says in particular that, if tl is an orthogonal representation of K, then the algebras A_ and R(Y) of operators on H are adjoint with respect to a fixed KIR­invariant positive-definite Hermitian structure B on H.

4.2. Let tlp C tl be the p-eigenspace of 'lrh. Recall I from (3-27) and notice

that I E ~Z+ + ~.

4.3 Lemma. The spectrum of 'lrh on tl is {l, l - 1, ... , -l} so that

tl = tll + tll-l ... + tl_1 (4- 2)

Differential Operators 79

is a decomposition into 2l + 1 non-zero eigenspaces. We have

tll = Cz and tl-l = Cz (4- 3)

and (pSP(Ll»(Z) = tll_p where p E Z+.

Proof. Since tl ~ V-y ~ V';, the highest and lowest eigenvalues of trh on tl are ±')'(h). But P is a weight vector of weight 2')' by 3.8 and plainly [h,P] = 2lP by (3-27) so that ')'(h) = l. The rest follows easily from (3-7) and 3.6. •

4.4. Suppose tl is an orthogonal representation of K, i.e., tl admits a com­plex symmetric K-invariant bilinear form. Then tl is the complexification of a real Kit-module tlR; let j' : tl --+ tl be the corresponding complex con­jugation map. Since tl is a complex representation of e we clearly have j'(x . v) = j(x) . j'(v) if x E e and v E tl. Now since h is pure imaginary we have j(h) = -h and consequently j'(tll) = tl_l. It follows then because of (4-3) that, after rescaling z if necessary, we have j'(z) = z. Now the transpose of j' defines a real structure on tl* and then we observe that Y C tl* is stable under the corresponding complex conjugation map on tl*, Indeed, clearly complex conjugation permutes the K-orbits on tl* and so j'(Y) = Y since z,j'(z) E Y. Thus j' defines a complex conjugation mapping j' : R(Y) --+ R(Y). From now on we write 9 = j'(g) where 9 E R(Y).

4.5. Theorem. Assume tl is an orthogonal representation of K. Then H admits a Kit-invariant positive-definite Hermitian inner product B such that, for every g E R(Y), multiplication by 9 and Dg are adjoint operators on H with respect to B, i. e.,

B(gs, s') = B(s, Dgs') (4 - 4)

for all s, s' E H. Furthermore B is unique up to rescaling by any positive real number.

Proof. By 2.12 we may choose a K-invariant E-lift E' on CT such that the spectrum of E' on H is Z+ so that

(4- 5)

We define B(s,s') = 0 if s E Hp and s' E Hpl with P =1= p'. Then B will be the sum of components Bp, p E Z+, where Bp is a positive-definite Hermitian inner product on Hp.

80 R. Brylinski and B. Kostant

We will construct the pairings Bp by induction on p in the following way. Now Hp is a simple K-module by 2.12 and hence Hp admits a K R-

invariant positive definite Hermitian inner product Dp which is then unique up to rescaling by a positive real number.

If V is a complex vector space then let V denote the complex conjugate complex vector space to V, i.e., V has the same underlying real vector space structure but the complex linear structure is modified by complex conjugation. Notice that if D is a Hermitian form on V then the map D : V x V ---+ C is complex bilinear and consequently defines a complex linear map jj : V ® V ---+ C. If V is a KR-module then V has a natural KR-module structure and V ~ V* as KR-modules.

Let

Bo =Do (4 - 6)

Now suppose we have defined a KR-invariant positive-definite Hermitian inner product Bp on Hp where p E Z+. We claim that the following equation defines another such form Bp+l on H p+1 :

(4- 7)

where v E U, we recall (2-7) and s E Hp while s' E H p+1'

The first issue is to show that (4-7) determines a well-defined pairing Bp+l on H p+1' Since Rl(Y)Hp = Hp+1 by 2.9, the vectors of the form vs ® s' span Hp+1 ® Hp+1 Therefore the problem is to show that (4-7), extended additively, assigns a unique value to Bp+1 (t, t') for any pair t, t' E

Hp+1' We may form the following diagram in which all four maps are complex

linear and KR-linear:

'" --+

(4 - 8)

where 0: and (3 are given by o:(s®v®s') = s®D:;;s' and (3(s®v®s') = vs®s' while e will be defined momentarily.

Now 0: and (3 are non-zero by (3-18) and 2.9(iii). To show that (4-7) defines B p+l, it suffices to show that if w E Hp ® U ® H p+l then

(4 - 9)

We now observe that we can define e so that the diagram is commutative.

Differential Opemtors 81

Indeed Bp 0 0: and Dp+1 0 f3 both define KR-linear maps

In addition both these maps are non-zero since 0: and f3 are surjective; indeed the surjectivity of the maps u 0 H p+1 -+ H P' v 0 S' t--+ Dvs', and IIp 0 u -+ IIp+1. s 0 V t--+ VS, follows from representation theory since each IIp is a simple KR-module. But using 2.12 we find

as Kit-modules and consequently (Hp 0 U 0 II p+d KR = C. Thus the maps Bp 0 0: and Dp+1 0 f3 are the same up to a non-zero scalar factor; equivalently, there exists a unique scalar cp+1 E C· such that if we de­fine ep+1 = Cp+1Dp+1 then the diagram (4-8) is commutative. Now (4-9) follows easily. This proves that Bp+1 is well-defined and furthermore

(4 - 10)

Now Bp+1 is positive-definite Hermitian if and only if cp+1 is positive real. To verify Cp+1is positive real it suffices to show that Bp+1 (t, t) is a positive real number for some section t E H p+1' Let s' E Hp be a non-zero highest weight vector for the f-action. Then (1TX)(S'Z) = S'(1TX)(Z) if x E fl. Using this we find

1 1 Dz(s'z) = -(1TP)(S'Z) = -s'(llP)(z) = s'

z z (4 - 11)

since (1TP)(Z) = z by 3.8. But then (4-7) gives

(4 - 12)

Thus Bp+1 (s'z, s'z) is positive real. Using the isomorphism (3-15), we easily obtain (4-4) from (4-7). Thus

we have constructed the desired form Band B is completely determined in (4-6) by Bo. It follows easily that B is unique up to scaling. •

4.6. Remark. Let A = E(E) be the symbol of the Euler vector field so that A E R[l](T*Y) (see A.10.). Let M c T*Y be the open dense set where A is non-zero. Let A_ C R(T*Y) be the K-invariant (Poisson abelian) subalgebra generated by the symbols of the elements of A_ (3.10). Then, because of (2-3), corresponding to A_ we have a K-invariant morphism

82 R. Brylinski and B. Kostant

tv T*Y --+ Y. We will show in a subsequent paper that tv defines a K-invariant polarization

tv: M --+ Y (4 - 13)

and furthermore the two polarizations (3-23) and (4-13) are transverse over M. Moreover we will show that R(Y), A_ and A-1 generate R(M) and using this we will deduce generators for a certain localization of V(Y, CT )

which is then a ring of pseudo-differential operators on sections of CT. We will also show that in the context of the geometry of M, the symbol

of D z arises quite naturally; indeed this motivated our construction of D z .

5. Local coordinates on Y

5.1. In this section we introduce a set of local coordinates (see A.14) on Y. We then compute D z and some other quantities explicitly in terms of these local coordinates. Finally we sketch how to use the generalized Capelli identity of Kostant and Sahi in order to explicitly compute the action of Dz on H. In a subsequent paper we will carry this out in the case where CT is a half-form bundle on Y.

5.2. Proposition. Let V1, ••. , Vn be a basis of tJl-1 (see (4-2»; then

n = dim t1 = dim tJl-l (5- 1)

The (n+ 1) functions Z, Vb • .. , Vn E R1 (Y) form a local system of algebraic coordinates on Y (see (A.14». In fact, if yo C Y is the open set where Z is non-vanishing, then these local coordinates define an isomorphism of

varieties

yo --+ C* x Cn , (5- 2)

and we have

(5 - 3)

In particular Y is a rational variety and

dimY = n+ 1 (5-4)

Differential Opemtors 83

Proof. It follows easily using the cell decomposition of JP>(Y) that

s.(, = yo (5- 5)

where (, is as in 2.4 and S c K is the subgroup corresponding to the Lie subalgebra

(5 - 6)

so that S = (expeh) . Kl (notation as in 3.2). Furthermore the isotropy group Se is finite.

Let n = dim tl = dim Ll and let Xl,'" ,xn be a basis of t l . We claim that the morphism

n e* x en ____ yo, (c, tl, ... , t n ) f-> (exp L tiXi) . (c(,) (5 - 7)

i=l

is an isomorphism. In fact we next construct the inverse map to (5-7). Using the fact that tl is a direct sum of simple to-modules corresponding to the simple factors of t, we find that the map tl ®Ll ---- e, x®y f-> 'Y([x, y]}

is a perfect pairing. Since tll-l = (7TLt}(z) by 4.3, it follows that the map tl®tll-l ---- tll, x®v f-> (7TX)(V), is a perfect pairing and dim tll-l = n. Thus, there exists a unique basis Vl, ... ,Vn of tll-l such that, for 1 :::; i, j :::; n,

(5 - 8)

After possibly rescaling (" we have < z, (, >= 1 where <, > is the natural pairing of tl with tl*. Then < Vi, Xj . (, >= -{ii,j' But then the functions z, -vdz, ... , -vn/z E R(YO) define the inverse morphism to (5-7). It follows now that (5-2) is an isomorphism. The second equality in (5-3) is now clear. We find using 2.5 that yo is the open set of the affine variety Y where z is non-vanishing. Thus R(YO) = R(Y)[1/z] = R(Y)[1/z]. •

5.3. From now on we adopt the notations of 5.2 and its proof, in par­ticular, < z, (, > = 1 and the bases Xl,"', xn of tl and Vl,"', Vn of tll-l satisfy (5-8). But then (5-8) and (3-20) imply that the (unique) expression for 7TXi in terms of our local coordinates is

where i = 1, ... ,no

a 7TXi = Z-a

Vi (5- 9)

84 R. Brylinski and B. Kostant

5.4. Remark. Using 5.3 we can give another proof of 3.14.

5.5. Clearly we have

(5 - 11)

5.6. Now we have a Ko-linear isomorphism

LI ® til --+ til-I, y ® v t-t [y, v] (5 - 12)

It follows that we have a Ko-linear graded algebra isomorphism

(5 - 13)

where y = [y, z] if yELl. Notice that if a E SpeLl) then the ~-weights satisfy

weight a = PI + weight a (5 - 14)

Similarly we have a graded algebra isomorphism

(5 - 15)

where xCv) =< [x,v],," > if v E tll-I and x E t l . But every ¢ E S(tJi_l) may be regarded in a natural way as a constant coefficient differential operator 8¢ on S(tll-l). Thus we obtain a graded algebra isomorphism

8 8 S(td --+ C[-8 , ... , -8 ],

VI Vn b t-t 8b (5 - 16)

Now (5-9) immediately gives

5.7. Proposition. The expression for Dz in our local coordinates is

5.B. Lemma. The expression for z in our local coordinates is

if z=-­z21-1

(5 - 17)

(5 - 18)

Differential Operators 85

where N E S21(Ll) is a Ko-semi-invariant of weight X- 2 •

Proof. By (5-3) and the complete reducibility of the Ko-action on S(t1)

we have z = zPii where a E St(Ll)O: and p E Z. Then by degree we have 1 = p + t. By weight, we have -"( = P'Y + t"( + a because of (5-14) and thus -2"( = a. But then because of 3.6 and (3-27) it follows that t = 21 and so p = 1 - 21. The lemma is now clear. •

5.9. Now zDz E 1)o(Y, Cr ) is a Ko-invariant degree 0 differential operator. It follows then using 2.12 and 3.11 that if p E Z+ then zDz acts on each simple Ko-submodule of H[v+p''Yl by a scalar. This proves

5.10. Lemma. Suppose that F C H[v+P'Yl' P E Z+, is a simple Ko­submodule. Then there exists a scalar CF E C such that for all s E F we have

(5 - 19)

5.11. Assume we are in the situation of 5.10. Notice that, since Dz(H) C

H by 3.10, we must have CF = 0 in (5-19) unless Fe zH. We will now sketch how to use the generalized Capelli identity of

Kostant and Sahi ([K-S)) to compute the scalar CF. This identity com­putes (in particular) the scalar by which the Ko-invariant differential op­erator N(8P) acts on any simple the Ko-submodule of S(Ll)'

Let So E H[vl be as in 2.9(ii). It follows from the description (5-5) of yo that So has no zeroes on yo. Thus s/so E R(YO) if s E H. It follows now using 5.2 that F = zP F*so where F. C S(L1) is a simple Ko-submodule and p E Z. Now (5-17) and (5-18) give

zDz = N(8F) (5 - 20)

We then find, for a E F., that

(5 - 21)

where the first equality follows using (3-13) and the fact P E S(tt}. But

easily we find that N(8F)(ii) = N{aP)(a). We conclude that CF is equal to the scalar by which the operator N(8P) acts on F. and hence is computed by the generalized Capelli identity.

86 R. Brylinski and B. Kostant

Appendix. Differential operators on algebraic varieties

A.I. We present here a summary of the basic notions concerning algebraic differential operators. We give the main constructions and give proofs of a few key results that have been used in the paper. Differential operators define a sheaf; in particular a differential operator on a variety X determines a differential operator on every open subset of X. The basic reference here is IGr]; we note that the context in IGr] is vastly more general than what we consider here. Also see, e.g., IHa] or IMum] for the theory of differentials and the tangent sheaf.

A.2. Let A be a complex commutative algebra and let M be an A-module. Let Z+ be the set of non-negative integers.

If m E Z+ let ;om(A,M) be the space consisting of D E EnddM) which satisfy

lam, ... lal. lao, Dj]· .. ] = 0 (A -1)

for all ao, ... ,am E A. Here we regard a E A as a multiplication operator on M. We set ;om(A, M) = 0 if m < O.

Then ;o(A, M) = UmEz+;om(A, M) is a subalgebra of EndcM called the ring of differential operators of the A-module M. We say D E ;o(A, M) has order m if D E ;om(A,M) but D ¢ ;om-1(A,M). So ;om(A,M) is the space of differential operators of order ::; m. The order filtration {;om (A, M)}mEZ+ of ;o(A, M) has the three properties

(i) ;oO(A, M) = A if M is cyclic as an A-module (ii) ;om (A, M);on(A, M) c ;om+n(A, M) (iii) l;om(A, M), ;on(A, M)] c ;om+n-1(A, M) if M is cyclic Thus, if M is cyclic, ;o(A, M) is a Lie algebra under the commutator

and each space ;om(A, M) is a ;o1(A, M)-module under the adjoint action and an A-module under multiplication. The associated graded space

(A - 2)

inherits the structures of a graded commutative algebra, graded module over ;01 (A, M)/;oO(A, M) and a graded A-module.

If M = A (with the multiplication action) then we write ;om(A, A) = ;om (A) and then ;o(A, A) = ;o(A) is the ring of differential operators of the algebra A. Then property (i) implies that ;oO(A, A) = A. We have a natural direct sum of A-modules ;o1(A) = der(A) EI1 A where der(A) is the Lie algebra of derivations of A. If A = C[X1' ... ,xp] is the polynomial

Differential Opemtors 87

algebra then we obtain the Weyl algebra

(A - 3)

A.3. We assume (for convenience) from now on that A is an integral domain and the A-module M is torsion-free. If b E A with b =f 0 then let A[b-l ) be the localization of A at b and let N[b- l ) = N ®A A[b- l ) where N is an A-module. Let N' = N ®A A' where A' is the fraction field of A. If N is torsion-free then the natural maps N - Nb - N' are injective and A-linear and we may regard them as inclusions.

A key lemma says that there is a unique extension of D E f)m(A, M) to a differential operator Db E f)m(A[b- 1), M[b- 1)) of the same order. This lemma follows easily by induction on the order of D and then we obtain the inductive formula

D (~) = D(s) - [D,b]b(g) b b b (A- 4)

where s E M. The resulting map f)(A, M) _ f)(A[b- 1], M[b- l )), D 1--+ Db, is an injective ring homomorphism; we may regard this as an inclusion and then simply write D for Db.

FUrthermore the construction in (A-4) defines an algebra inclusion f)(A, M) c f)(A', M'). The key step in verifying this is to check using induction that (Db)c = Dbc if b, c E A are non-zero. Then if s, t E M are such that sib = tic in M' we find Db(g) = (Dch(~) = Dbc(~) =

(Db)c(~) = Dc(~). A.4. Let X be an irreducible complex algebraic (quasi-projective) variety with structure sheaf Ox. Let R( Z) denote the algebra of regular functions on any variety Z. Let:F be a quasi-coherent sheaf of Ox-modules. We assume :F is torsion-free, i.e., :F(U) is a torsion-free module over R(U) = Ox(U) if U c X is open affine. Then the restriction maps on sections of :F are injective and we may regard them as inclusions. For instance, :F may be the sheaf of sections of an algebraic vector bundle F _ X (see, e.g., [Ha, pp. 128-9] or [Mum, pp. 208-15)). Our main interest here is the case where :F = C and C is the sheaf of sections of a line bundle L - X. Then C is a locally free sheaf of rank 1, or, equivalently, C is invertible. However much of theory works in exactly the same way for the more general sheaf :F.

For each m E Z+, X admits a unique sheaf VX.:F of Ox-modules such that if V cUe X are affine open sets then

VX.:F(U) = f)m(R(U),:F(U» (A - 5)

88 R. Brylinski and B. Kostant

and the restriction homomorphism puv : V'X,:F(U) -+ V'X,:F(V) is defined in the following way. Let {UdiEI be an affine open cover of V where Ui = Uc/>;, ¢i E R(U) and Uc/> denotes the non-zero locus of ¢ E R(U). Then we have natural identifications R(Ui) = R(U)[¢i-1j and F(Ui) = F(U)[¢i-1j (here we are using the fact that F is quasi-coherent). Now if D E '1)m(R(u),F(U» then we have defined in A.3 the extension Di E

'1)m(R(ui),F(Ui» where Di = Dc/>i. If s E F(V) then let Si = slu; so that Di(Si) E F(Ui). But Si = ti/¢i where ti E F(U) and over Uij = Ui n Uj = Uc/>;c/>j we have td¢i = tj/¢j. It follows using A.3 that the sections Di(Si) and Dj(sj) coincide over Uij for all i,j E I. Thus the family {Di(Si)}iEI defines a section s' E F(V). It follows easily that {DihEI defines a unique operator D' E '1)m(R(V),F(V» such that D'(S) = s' for each s. We now define puv(D) = D' .

It is routine to verify that the map puv is independent of the choice of cover {UihEl and that furthermore the data we have given defines a (unique) sheaf V'X,:F of Ox-modules where Ox acts by left multiplication.

Suppose V cUe X are open subsets. We have a natural identifica­tion V'X,:F(V) = VU,:Flu (V) and so we may write vm(v, F) = V'X,:F(V). Furthermore it follows easily using A.3 that for all open sets V C U the restriction maps Puv are injective and thus we may regard them as inclu­sions. In other words, a differential operator D defined over an open set U is uniquely determined by its restriction to any open subset V.

Now a section D E vm(U,F), is specified by the data {(Di , Ui)}iEI where {UihEl is an affine open cover of U, Di E '1)m(R(Ui),F(Ui» and Di and Dj coincide over Uij = Ui n Uj for all i,j E I. The last con­dition means that Di and D j extend to the same differential operator Dij E '1)m(R(uij ),F(Uij» (indeed Uij is affine open as it is the inter­section of two such sets). So if S E F(U), Si = slui and Sij = SIUij then Di(Si)lu;j = Dij(Sij) = Dj(sj)luw Consequently the family of sections {Di(Si)}iEI defines a section (which we will call) D(s) E F(U); moreover D(s) is independent ofthe choice of {UihEl. This shows that, as one would hope, differential operators act on sections of F. In fact we obtain a natural linear map

(A - 6)

In general (A-6) is not injective. By construction (A-6) is an isomorphism if U is affine; we will show in A.6 below that this also holds if U is quasi-affine.

A.5. The union

(A -7)

Differential Opemtors 89

is, in the obvious way, a sheaf of Ox-modules and a sheaf of non­commutative rings on X called the sheaf of differential operators of F. We write V(U,F) = VX,:F(U) where U c X is open. If F is the sheaf of sections of F as discussed in A.4 then we may call VX,:F the sheaf dif­ferential operators on sections of F; we then write VX,F = VX,:F and V(U, F) = V(U,F).

We say D E V(U,F) has order m if D E vm(U,F) but D ¢. vm-l(u,F). It follows since X is connected and F is torsion-free that D has the same order on every open subset of U. The order filtration {1)~,:F }mEZ+ of VX,:F satisfies all the corresponding properties given in A.2. In particular .c satisfies

vt.c = Ox

and we have V~,:FVX,:F c V 1]t}n and

[vm vn j V m +n- 1 X,.c, X,.c C x,c

(A - 8)

(A - 9)

We write V~ = V~,ox and Vx = Vx,ox and also vm(x) = V~(X) and V(X) = Vx(X). Then Vx is the sheaf of differential operators on X.

A.6. If U is quasi-affine then (A-6) is an isomorphism; in particular then we have a natural linear inclusion

V(U, F) c End r(U, F) (A -10)

To prove this we will construct the inverse map to (A-6). Since U is quasi­affine, we have an open embedding U C U' where U' is an affine variety. Then U' - U is a closed set in U' and consequently is the zero-locus of a finite set of (non-zero) functions ¢>1, ... , ¢>r E R(U'). Then VI ... , Vr is an affine open cover of U where Vi = {x E U'I¢>i(X) = O}. Of course, by restriction, ¢>1, ... , ¢>r E R(U).

Now R(Vi) = R(U')[¢>i-1 j for each i and then, since R(Vi) :J R(U) :J

R(U'), we obtain R(Vi) = R(U)[¢>i-1 j . Furthermore we claim that

(A - 11)

i.e., the natural map of R(Vi)-modules ai : F(U)[¢>i-1 j -+ F(Vi) defined by the restriction homomorphism F(U) -+ F(Vi) is an isomorphism. Indeed we have F(Vi) = F(U')[¢>i- 1 j and so it follows that ai is surjective. On the other hand if t E F(U)[¢>i-lj then ¢>ft E F(U) for some positive integer n. Now if ai(t) = 0 then ¢>ft vanishes on Vi. But then, since F is torsion-free,

90 R. Brylinski and B. Kostant

¢it vanishes on U. Therefore t = O. Thus 0i is injective and we have established (A-ll).

The construction of A.3 now applies so that given D E 'D(R(U), F(U» we obtain Di E 'D(R(Vi),F(Vi» for i = l. .. ,r. It is easy to check that these Di define a section D' E V(U, F) and that D' maps to D under (A-6).

Notice that the argument is more simple in the case where F = Ox.

A.T. Assume X is quasi-affine. Suppose A E vm(x, F) is a differential operator of order m and f E R(X) with A, f =I- o. We observe that if there exists a linear transformation D E End r(X, F) such that A = f D then D E vm(x, F) and D has order m.

We will prove the observation. Let go, ... , gm E R(X). Let U c X be the subvariety where f =I- O. Then A defines an order m differential operator on U and, over U, we have f- 1 A = D. Thus, working over U we find

It follows that D E vm(X,F). Furthermore if D E V m- 1(X,F) then similarly we find that A E V m - 1 (X, F), which is false.

A.S. The tangent sheaf Tx on X is the subsheaf of Vi such that Tx (U) = der(R(U» (see A.2) for every affine open set U C X. Then Tx is a sheaf of Ox-modules and we have a direct sum of Ox-modules

vi =Tx EBOx (A - 12)

Also [Tx, Tx 1 c Tx. A section of Tx over an open set U C X is called an (algebraic) vector field on U. Thus a vector field on U is in particular an order 1 differential operator on U. The vector fields form a Lie algebra under the commutator.

If an algebraic group K acts on X then differentiation defines a Lie algebra homomorphism t -> Tx(X) where t is the Lie algebra of K. Given any Lie algebra a, we call a Lie algebra homomorphism a -> Tx(U) an infinitesimal action of a on U. More generally, we call a Lie algebra homo­morphism a -> Vi (U, F) an infinitesimal action of a on F over U.

A.9. Assume X is smooth. Then Tx is a locally free sheaf of rank d where d = dimX. The d-dimensional (algebraic) vector bundle on X associated to Tx is the tangent bundle TX -> X. Then Tx is the sheaf of sections of T X. The dual vector bundle to T X is the cotangent bundle

T: T*X -> X (A - 13)

Differential Opemtors 91

and its sheaf of sections is the cotangent sheaf T;. Notice that (A-13) defines a natural inclusion of algebras R(X) C R(T* X). Regular sections of the nth exterior power vector bundle I\. nT* X, 0 :::; n :::; d, are called (algebraic) differential n-forms on X. The canonical bundle on X is the line bundle I\.dT* X --t X. We have Txlu = Tu and Txlu = Tu if U c X is open.

Suppose x E U C X where U is an affine open set. Let mx C R(U) be the maximal ideal of functions vanishing at x. Let IX : mx --t mx/m;, be the natural projection. Then the associated bundle construction pro­vides an identification of the tangent space TxX = Tx (U)/mxTx (U). Now the formula (oxD)(/x¢) = (D¢)(x), ¢ E mx, defines a linear map Ox : der(R(U» --t (mx/m;)* since if D E der(R(U» then D(m;) C mx. Moreover the kernel of Ox is equal to mxder(R(U». Thus we obtain an isomorphism TxX --':::"(mx/m;)* and consequently an isomorphism for the cotangent space

m /m2--':::"T*X x x x (A - 14)

We may regard both these isomorphisms as identifications. Let U C X be an open set. Now if ¢ E R(U) then the differential of

¢ is the I-form defined at each point x E U by (d¢)x = IX(¢ - ¢(x». If ¢o, ... , ¢n E R(U) then ¢Od¢ll\.· .. 1\. d¢n defines an (algebraic) n-form on U. Furthermore, locally, n-forms of this type span the vector space of all n-forms. In fact if x E U then there exists an affine open set V C U with x E V and d functions '¢1, ... ,'¢d E R(V) vanishing at x such that Tv is a free Ov-module with basis d,¢1.' .. ,d'¢d'

If "I is a vector field on X then the Lie derivative of "I is the (unique) differential operator £7] E Vl(X, I\.nT* X) such that if U c X is open and ¢o, ... , ¢n E R(U) then

£7] (¢Od¢l I\. ... 1\. d¢n) = ("I¢O)d¢ll\.···1\. d¢n

+¢Od("I¢l)l\.d¢l I\. ... I\. d¢n + ... + ¢Od¢l 1\ •.• I\. d("I¢n) (A - 15)

The exterior derivative d : r(X,l\.nTx ) --t r(X,l\.n+1Tx ) is the (unique) linear map such that if U and ¢o, ... ,¢n are as above then

d( ¢Od¢l I\. ... I\. d¢n) = d¢o I\. d¢l I\. ... I\. d¢n (A - 16)

The exterior derivative defines a linear map d : I\nT; --t I\.n+1T;. The graded symmetric algebra

(A -17)

92 R. Brylinski and B. Kostant

is the pushdown sheaf T*OToX so that qu, sm(Tx» = R[m)(T*U) where R[m) (T* U) c R(T* U) is the subspace of functions which have homogeneous degree m under the scaling action of C* on the fibers of (A-13). Then R(X) = R[o)(T* X) and we have an algebra grading

R(T* X) = ffimEZ+ R[m) (T* X) (A - 18)

A.IO. Assume X is smooth. From now on we will only be concerned with the sheaf .c of sections of a line bundle L over X.

If U c X is open then the order m symbol map

Eu : 1)m(u, L) --+ R[m) (T*U) (A - 19)

is the unique linear map such that the values of Eu(D) on T;U satisfy

(A - 20)

where V c U is open, x E V, ¢> E R(V) with ¢>(x) = 0 and s E r(V,L). Then (A-19) is a linear map of R(U)-modules. We may write Em = Eu. Notice that if ¢> E R(X) then E(¢» = ¢>j if ~ is a vector field on X then E(~) = ¢>f. where ¢>f. E R(1) (T* X) is the obvious function defined by ~.

It is routine to verify that (A-20) defines an Ox-linear map, the order m symbol map on sheaves,

(A - 21)

Notice that the order 1 symbol map is, using (A-12), just the natural projection Tx ffi Ox --+ Tx. We then have the Ox-linear short exact sequences

(A - 22)

where m E Z+. On the level of sections (A-22) gives the exact sequence

(A - 23)

Thus we obtain, in the obvious way, a graded (degree 0) Ox-linear isomorphism of sheaves of algebras, the graded symbol map on sheaves

E: grVx,L~S(Tx) (A - 24)

Differential Opemtors 93

where gr'Dx,L = Ef)mEZ+ 'Dj(,dVX,[,l. Furthermore (A-23) defines an in­jective, graded (degree 0) algebra homomorphism, the gmded symbol map

Eu : grV(U,L) ~ R(T*U) (A - 25)

where we define

(A - 26)

The symbol map is the map

Eu : V(U, L) ~ R(T* U) (A - 27)

defined by Eu(D) = Eu(D) if D has order m. We may write E = Eu. Then E is not additive but does satisfy E('l/JD) = 'l/JE(D) if'I/J E R(U). Under the symbol map, the commutator of operators goes over into the Poisson bracket of symbols in the following way. The cotangent bundle T* X has a canonical symplectic structure. Then R(T* X) becomes a Poisson algebra under the corresponding Poisson bracket {, }. We have

{R[m](T* X), R[n](T* X)} C R[m+n-l](T* X) (A - 28)

Suppose Dl,D2 E V(X,L) are differential operators of orders m and n respectively. Then

(A - 29)

if [Dl' D2J has order m + n - 1, while {E(D1), E(D2)} = 0 otherwise.

A.II. Let X be a smooth affine variety. Then we have the well-known facts that the graded symbol map (A-25), where U = X, is surjective and furthermore the ring VeX) is generated by the functions on X together with the vector fields on X.

We note that no such easy characterizations of VeX) exist when X is affine and singular, when X is quasi-affine and smooth or when X is projective and smooth.

A.I2. Suppose that X is quasi-affine and a connected algebraic group K acts on X. Assume the line bundle L on X is K-homogeneous. Then clearly K acts as a group of algebra automorphisms of VeX, L) and e acts as a Lie algebra of derivations of VeX, L). The latter action is the representation

e ~ End VeX, L), (A - 30)

94 R. Brylinski and B. Kostant

where "Ix E VI (X, L), x E t, is the differential operator defined by differ­entiating the action of K on reX, L). Then (A-3D) defines a representa­tion U(t) ~ End VeX, L) where U(t) is the universal enveloping algebra. Clearly then both the K-action and the U(t)-action on VeX, L) preserve the order filtration. We call these actions the "natural actions" and denote them by g. D and U· D where 9 E K, u E U(t) and D E V(X,L). Re­call that an action of a group or an associative algebra on a vector space is called locally finite if every vector lies in a finite-dimensional subspace stable under the action.

We observe that the action of K on VeX, L) is locally finite and fur­thermore the differential of this K-action is the t-action defined by (A-3D). In particular then the natural actions of K and U(t) leave stable the same subspaces of V(X,L). If K is reductive, then it follows that moreover VeX, L) is a completely reducible K-module and decomposes into the al­gebraic direct sum of finite dimensional simple K-submodules.

We will prove the observation. First we show that the action of U(t) on VeX, L) is locally finite. We will prove by induction on m that the U(t)­action on each space vm(x, L) is locally finite. If m = D then VO(X, L) = R(X) by (A-8) and so U(t) acts locally finitely. Next suppose that U(t) acts locally finitely on vm-I(x, L), m 2:: l.

Now we may assume without loss that X is smooth; indeed the smooth locus Xs C X is a K-stable open dense set and hence we have a K-linear inclusion VeX, L) c V(Xs, L). But then since X is smooth we have the exact sequence (A-23) where we set U = X. Now the natural action of U(t) on R(T* X) arises from the action of K on T* X and hence is locally finite. Thus we can apply the following lemma where B = vm(x, L): if 0 ~ A..!!......BLc is an exact sequence of U(t)-modules and U(t) acts locally finitely on A and C then U(t) acts locally finitely on B.

Next we prove the lemma. We may regard a as an inclusion. Let bE B. We wish to show that U(t) . b is finite-dimensional. Now the space U(t)·{3(b) = Fee is finite-dimensional. Let J C U(t) be the annihilator of F. Then J has finite codimension on U(t); indeed we have an injective map U (t) / J ~ End F. Thus it suffices to show that J . b is finite-dimensional. But J·b C A since (3(J·b) = J·{3(b) = O. Furthermore, since grU(t) = Set) is Noetherian, we find that J is finitely generated as a left ideal of U(t) -let UI,' .. ,up E J be a finite set of generators and set ai = Ui • b. But then each space Ai = U(t) . ai C A is a finite-dimensional K-submodule and consequently J . b = L:f=1 Ai is finite-dimensional.

Thus U(t) acts locally finitely on V = VeX, L). Now let D E V. We next show that the K -submodule I C V spanned by K . D is finite­dimensional. Let A = U(t) . D so that A C V is a finite-dimensional

Differential Operators 95

K-submodule. We know that K acts locally finitely on the space r = r( X, L). Let {Bi hEZ+ be an increasing filtration of r by finite-dimensional K -submodules. Let J.l : V ® r ~ r be the natural map and let Ci = J.l(A ® B i) so that Ci c r is a finite-dimensional K-submodule. Let Ai be the image of the natural map l¥i : A ~ Hom(Bi , Ci). Since we have the inclusion A C End r by A.6, it follows that there exists j E Z+ such that l¥i is injective if i ~ j.

Let 9 E K. We have (g . D)(Bi) C Ci, i E Z+, since (g. D)(Bi) = g.(D(g-l.Bi )) = g·(D(Bi)) c g,Ci = Ci . Thus for each i we have a natural K-linear map Ui : I ~ Hom(Bi, Ci)j let ui(E) = Ei if EEl. But then using the K-action on Hom(Bi, Ci ) we find (g·D)i = g·Di E U(t)·Di C Ai. It follows that there exists a unique element a E A such that (g·D)i = l¥i(a) for all i ~ j. We easily conclude that g. D = a. This proves that I C A. It follows that I = A and the t-action on A is the differential of the K -action. We have now proven the observation.

A.13. If e is a non-zero vector field on a smooth variety X then a e-lift on L is an order 1 differential operator e' E VeX, L) satisfying the equivalent conditions

(i) e'(cps) = e(cp)s + cpe'(s) if U c X is open, cp E R(U) and s E r(U, L) (ii) B(e') = B(e) It is easy to verify that if e is an e-lift on L then so is e' + 1/J, where

1/J E R(X), and moreover every e-lift on L is of this form. On the other hand, if DE VeX, L) has order 1, then B(D) = cpe where e is a non-zero vector field on X and then D is an e-lift on L.

A.14. We will call a set of n rational functions ft, ... , In on X a local (algebraic) coordinate system on X if and only if there exists an open set U C X such that ft, ... , In are defined on U and the morphism I : U --. (Cn,

I(x) = (ft(x), ... ,In(x)), is an open embedding of the variety U. Ifwe can choose U = X then we call ft, ... , Ina coordinate system on X.

References

[B-K1] R. Brylinski and B. Kostant, Nilpotent orbits, normality and Hamil­tonian group actions, JAMS 7 (1994), 269-298.

[B-K2] R. Brylinski and B. Kostant, Minimal representations of E6 , E7 and Es and the generalized Capelli Identity, Proc. Natl. Acd. Sci. USA 91 (1994), 2469-2472.

[B-K3] R. Brylinski and B. Kostant, Minimal representations, geometric

96 R. Brylinski and B. K ostant

quantization and unitarity, Proc. Natl. Acad. Sci. USA 91 (1994), 6026-6029.

[Gar] D. Garfinkle, A new construction of the Joseph ideal, MIT Doctoral Thesis, 1982.

[Gr] A. Grothendieck, Elements de geometrie algebrique, EGA IV, Etude locale des schemas et des morphismes de schemas (quatrieme partie), Publ. Math. IHES 32 (1967).

[Ha] R. Hartshorne, Algebraic Geometry, Springer Verlag, New York, 1977. [He] S. Helgason, Differential Geometry, Lie Groups and Symmetric Spaces,

Academic Press, New York, 1978. [K-S] B. Kostant and S. Sahi, The Capelli identity, tube domains and the

generalized Laplace transform, Adv. Math. 87 (1991), 71-92. [Mum] D. Mumford, The Red Book of Varieties and Schemes, Lecture Notes

in Math. 1358 (1988), Springer Verlag, New York. [V-P] E.B. Vinberg and V.L. Popov, On a class of quasihomogeneous affine

varieties, Math. USSR Izv. 6 (1972), No.4, 743-758.

Department of Mathematics, Pennsylvania State University, University Park, PA

16802 and (visiting) Harvard University, Cambridge MA 02138

Department of Mathematics, Massachusetts Institute of Technology, Cambridge,

MA 02139

Received June 28, 1994

Groups and the Buckyball

Fan R. K. Chung, Bertram Kostant and Shlomo Sternberg

1. Introduction

T he purpose of this article is to collect a number of remarkable group theoretical facts having to do with icosahedral symmetry. Some of these have been already applied to the discovery and identification of the new C60 carbon molecule, called the Buckminsterfullerene, or buckyball, for short, and we hope that other results described here will find applications in physical properties of these molecules. We begin with a description of the molecule.

Take the regular icosahedron. It has twelve vertices with five edges emanating from each vertex, so thirty edges and twenty faces. If we truncate each vertex so as to get a pentagon, each of the (triangular) faces of the icosahedron becomes a hexagon. (They become regular hexagons if we truncate each vertex one third of the way along each edge.) Doing so, we obtain a figure with 60 vertices, each vertex has three emanating edges, two of which lie in pentagonal directions and the third is an edge of the hexagon. This is the buckyball. The icosahedral group I is the group of rotational symmetries of this figure which has 60 vertices, twelve pentag<:mal faces and twenty hexagonal faces, so 32 faces. It has 5 x 12 = 60 pentagonal edges (the pentagons do not touch one another) and each hexagon has three edges which are not pentagonal, but shared with an adjacent hexagon, so 20 x 3/2 = 30 such edges, so 90 edges in all. The Euler formula can be easily checked:

32 - 90 + 60 = 2.

The group I acts transitively on the space of vertices of the icosahedron, and the isotropy subgroup of each vertex consists of rotations through an­gles 2~11". Thus, this subgroup on the buckyball is the isotropy group of a pentagon, and acts transitively on the vertices of the pentagon. So I acts simply transitively on the vertices of the buckyball. It is this fact that accounts for its most remarkable properties from a mathematical point of view.

Although I acts transitively on the vertices, it does not act transitively on the edges. In fact, each vertex, v, lies on exactly one pentagon, and hence two of the edges emanating from v are pentagonal and one is hexagonal.

98 F. R. K. Chung, B. Kostant, and S. Sternberg

In other words, every other edge of each hexagon is purely hexagonal in that it lies on two hexagons, while the other three edges lie on a hexagon and a pentagon. So I acts transitively on the set of hexagonal edges and on the set of pentagonal edges. In the simplified chemical model for C60

it is assumed that a carbon atom is placed at each vertex of the buckyball and that the pentagonal edges are single bonds while the hexagonal edges are double bonds. We will adopt this chemical language: we will call the pentagonal edges single bonds and the hexagonal edges double bonds.

The rotation, t a , which maps the vertex a into the vertex b connected to a by the double bond u emanating from a must map u into the unique double bond emanating from b, and this double bond is u. Hence tau = u and therefore tab = a. Thus t~a = a and since I acts simply transitively this implies that

t~ = id.

Similarly, if 8 a denotes the clockwise rotation which moves a to its nearest neighbor on the unique pentagon on which a lies, then

8~ = id.

In other words, motion along a single bond is a rotation through angle 211"/5.

We can ask, within the 60! element group of all permutations of the ver­tices (isomorphic to 860), which transformations commute with the action of I? This will be another sixty element group isomorphic with I. Indeed, once we fix some vertex, we can use the action to identify all the vertices with I, where I acts on itself by left multiplication. But the elements which commute with left multiplication on a group are right multiplication.

Call this commuting group R. Fix one vertex, say a, so we get an iden­tification of C60 with I under which a corresponds to the identity element, e. Let t E R be the element which corresponds to right multiplication by ta and similarly 8 = 8 a . So t carries a into the unique vertex joined to a by a double bond. But then, by the transitivity of the action of I, t must carry every vertex into the neighbor connected to it by a double bond. Similarly, the single bonds emanating from every vertex will correspond to either 8 or 8-1 . Now we can clearly walk from any vertex to any other by a succession of single or double bonds. This shows that 8 and t generate R and hence that 8 a and ta generate I.

If G is any group, and {g} is a set of generators of G then the associated Cayley graph is defined to be the graph whose vertices are the elements of G, and two vertices a and b are joined by an edge if a = gb or a = g-1b where 9 is one of the set of preferred generators. So what we have shown is that

Groups and the Buckyball 99

Theorem 1. C60 is a Cayley graph.

Having fixed a vertex, the above construction associates the generator t to every double bond, while our construction associates either s or s-1 to each single bond. But we can make a global choice as follows: Think of the buckyball as a polyhedron, that is, put in the faces. Then a choice of orientation of R3 determines an orientation of the surface of the buck­yball and hence of each pentagon. Since each pentagonal edge lies on a unique pentagon, the orientation of the pentagons induces an orientation of every pentagonal edge, so now every vertex has a unique pentagonal edge emanating from it. Now if s is the group element which moves a into its nearest neighbor along the pentagonal edge emanating from a, we have labelled every pentagonal edge by s.

Notice that the buckyball is also invariant under the central inversion through the origin; call this transformation P for "parity". We will let

the direct product of I with the group of central inversion through the origin. So h C 0(3) is the full group of orthogonal symmetries of C60

while I C 80(3) is the full group of rotational symmetries.

The group h is a trivial central extension of I; it is a direct product. From a mathematical point of view, there is a much more interesting central extension, which can be described as follows:

There is a homomorphism 71" : 8U(2) -t 80(3) which is a double cover, i.e. ker 71" = {e, -e}. We can then consider the 120 element group G = 71"-1 I. This group will play the hero's role in what follows.

2. Group identifications

In this section we describe the group isomorphisms

I rv A5 rv Pl(2, 5)

and G rv 8l(2,5)

as well as the remarkable Galois embedding

8l(2, 5) -t 8l(2, 11).

100 F. R. K. Chung, B. Kostant, and S. Sternberg

We begin with the isomorphism I '" As. For this, it is enough to find a five element set on which I acts effectively, since As is the only sixty element subgroup of 8 5 : Any double bond of the buckyball has a unique opposite double bond, and together they determine a plane. There will be exactly two pairs of of opposite bonds whose planes are orthogonal to these planes, and they will be orthogonal to each other. So the set of thirty double bonds decomposes into five classes of six elements each, i.e. five classes of mutually perpendicular "coordinate planes". This gives a homomorphism of I into the group of permutations of these five classes of coordinate systems, and it is easy to check that this is an isomorphism. We now turn to the relation with the general linear groups.

For any finite field F q with q elements, the group Gl(2, q) of all invertible two by two matrices with entries in F q has (q2 _1)(q2 -q) elements, because there are q2 - 1 choices for the entries in the first column and for each first column, the second column must be linearly independent, so q2 - q choices for the second column. Thus the number of elements in 81(2, q) is given by (q2 _ 1)(q2 _ q)/(q _ 1) = q(q2 - 1). For q = 5 we see that

1 81(2,5) 1= 120.

The group 81(2, q) acts on the projective space, P! of lines through the origin in the plane F~, and the kernel of this action consists of {id, -id}. If q is odd, this kernel contains two elements, if q is even, then -id = id. The corresponding quotient group, the group of projective transformations is known as Pl(2, q). If q = 5 the group Pl(2,5) has 60 elements and is isomorphic to A5. This can be seen as follows. There are (q2 -1)/(q -1) = q + 1 points on the projective line, P!. For our case, q = 5, we will identify these six points with the six lines through the centers of opposite pentagonal faces of the buckyball. (Or, what amounts to the same thing, to the six lines through opposite vertices of the original icosahedron.) Let us call these six lines pentagonal axes. The subgroup of the icosahedral group, I, fixing a given pentagonal axis is the dihedral group, D5 , consisting of rotations through angles 2k7r /5 about the axis, together with five rotations through 180 degrees in the plane perpendicular to the axis, and which interchange the two opposite pentagons. On the other hand, the subgroup of 8l(2, q) fixing a point in the projective line, say the point [(1, o)]f, is the Borel subgroup

B = { (~ a~l)}' a E F;, bE F q ,

of all upper triangular matrices in 8l(2, q). This subgroup has q(q - 1) elements, so 20 for the case q = 5. It is clear that B / {id, -id} = Ds . Now we identify pA = 8l(2, 5)/ B with the set of pentagonal axes which is 1/ D5 ,

Groups and the Buckyball 101

giving an action of P1(2, 5) on the buckyball and hence a homomorphism of P1(2, 5) into I which must be an isomorphism since P1(2, 5) is simple.

If we identify the six axes of the icosahedron with the six points of the projective line P~, we are in the following situation: A5 and PSL(2,5) are both identified as simple subgroups of order 60 of the permutation group S6. We must show that they are isomorphic. Since they are both simple subgroups, they are, in fact, subgroups of A6 • Now we can make use of the following assertion:

Let Band C be two subgroups of index 6 in A6. Then there is a ¢ E AuT(A6) such that ¢(A) = B.

Proof. Consider the two six element sets A6/ A and A6/ B. The group A6 acts as permutations of each of these six element sets. Let us choose an identification of A6/A with the set {1,2,3,4,5,6} so that A is identified with 6. This gives a homomorphism of A6 with a subgroup of S6; call this homomorphism, ¢A. Since A6 is simple, this is an injection; and since the only subgroup of index 2 in Sn is the alternating subgroup (for any n > 1), this is an isomorphism of A6 onto the alternating subgroup of S6. Similarly for A6/ B. Hence

¢ = ¢B o¢-;.l

is an automorphism of A6 with ¢(A) = B.

Every alternating group An has outer automorphisms coming from con­jugation by odd elements of Sn. But notice that the automorphism, ¢ given above is not of this type. Indeed, PS1(2,5) acts transitively on P~ while A5 fixes a point. Hence the isomorphism between PS1(2,5) and A5 given above illustrates a remarkable (and unique) property of A6 among all simple alternating groups:

Aut(A6) is bigger than S6.

An extension of the above argument shows that

G rv Sl(2, 5)

where, we recall, G is the double cover of I sitting as a subgroup of SU(2).

We should also mention that if we take q = 4, then Sl(2,4) = P1(2,4) has 60 elements and is also isomorphic to A5 • Indeed, there are five points in P~ which are permuted by the action of Sl(2, 4). This gives an isomor­phism of Sl(2, 4) onto a sixty element subgroup of S5 and hence onto A 5. To visualize this isomorphism on C60 , each class of coordinate planes is identified with a point of P~. So

I rv Sl(2,4).

102 F. R. K. Chung, B. K08tant, and S. Sternberg

In our identification of I with Pl(2, 5) we considered a pair of opposite pentagons as a point in pA. But we can consider the individual pentagons (Le. the vertices of the original icosahedron) as points of pit. It was proved by Galois (in a letter to Chevalier written on the eve of his fatal duel) that Pl(2,p) does not have a non-trivial permutation representation on fewer that p + 1 elements if p > 11. However for p = 2,3,5,7,11 there exist transitive permutation representations on p elements, and this p element set can be realized as a set of p configurations of pairs of points in P~. Here is a sketch of the construction. We refer the reader to the book by Conway and Sloane, [4], p. 268 for details. Let us label the twelve points of pit as

0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,00.

The group Pl(2, 11) acts on pit by fractional linear transformations:

ax+b x I--t ---.

cx+d

Let C denote the set of partitions of the above letters into six pairs, so a point of C will be a partition of the form

{a, b H c, dH e, fHg, h H i, jH k, l}

where a ... 1 is a permutation of the above twelve letters representing points of Pit. As the group Pl(2, 11) acts on Pit, it acts on C. In general, the orbit of an element of C will be quite large. But, it turns out that there is an eleven element orbit! Consider the configuration,

co = {00,0}{1,6}{3,7}{9,10},{5,8},{4,2}.

Along with the pair {oo, O}, this configuration contains in each pair exactly one non-zero square in Fll, namely 8 = 1,4,9,5,3 along with 68. In other words,

co={oo,O}U U {s,6s}.

sEF~~

Now the diagonal subgroup b. c 8l(2, 11) acts on Pll as fractional linear transformations by sending

and so b. preserves Co. Thus, if we let Ci denote the image of Co under the translation x I--t x + i, i E F ll , then the eleven element set

Groups and the Buckyball 103

is preserved by the Borel subgroup B. So much would be true for any odd prime, p, and 6 replaced by any non square in F p. However the following miracle occurs for the choice of p = 11 and CO as above: Let

be the Weyl group generator so

wx = _x- l

as a fractional linear transformation. Then CO, Cl and Cs are all preserved by w while

which can be checked by direct computation. For w2 = id as a fractional linear transformation, this shows that Cll is preserved by wand hence by all of 81(2, 11), since Band w generate all of 81(2,11). As the translations act transitively on Cll , we conclude that the 660 element group, Pl(2, 11) acts transitively on Cll, and hence the isotropy group of a point, say CO, has sixty elements. One checks that this subgroup is isomorphic to I. For example, within this isotropy group the subgroup that fixes the pair (00,0) is the image in Pl(2, 11) of the group generated by the diagonal matrices and w, hence the subgroup is isomorphic to D 5 • It is easy to find within the isotropy group an element of order three that does not centralize the D5 and this shows that the subgroup is isomorphic to I. The fact that the subgroup fixing the pair (00,0) is D5 shows that we should think of the pairs entering into the definition of Co as pairs of opposite pentagons (opposite vertices of the original icosahedron). Thus the geometry of the icosahedron is determined group theoretically by identifying its vertices with points of pil and its pairs of opposite vertices as forming a configuration like Co. The relation between the groups I and Pl(2, 11) is visible at the representation theoretical level when we compute the electronic spectra of the buckyball in the Huckel model as we let the strength of the double bond tend to zero. We will explain this in section 7. For this purpose we record the fact that under our embedding of Pl(2, 5) -t Pl(2, 11) the subgroup of Pl(2, 5) fixing a pentagon (say the point 00 of PIl) is a cyclic group of order five, and is, in fact, the group

of multiplication by squares (the action of diagonal matrices).

104 F. R. K. Chung, B. Kostant, and S. Sternberg

3. Icosahedral structures and Pl i •

The vertices and edges of an icosahedron define a graph with 12 vertices and 30 edges. A 12-element set Y will be said to have an icosahedral structure if 30 pairs of vertices are chosen so that, as edges, the resulting graph is isomorphic to the graph of an icosahedron.

Assume Y has an icosahedral structure 8. If y E Y then there is a unique point y', referred to as the antipode of y, such that any edge path joining y and y' has at least 3 edges. Of course y" = y. We refer to {y, y'} as an antipodal pair. If {y, y'} c Y is an antipodal pair and x E Y\ {y, y'} then either {x, y} is an edge or {x, y'} is an edge but both are not edges. If {x, y} is an edge we refer to x as a neighbor of y. Every y E Y has 5 neighbors. The 5 neighbors of y' are referred to as coneighbors of y. Of course being a neighbor or coneighbor is a reflexive relation. Any a E Perm Y defines another icosahedral structure a8 on Y where {x,y} is an edge for a8 if and only if {a-Ix, a-Iy} is an edge for 8 . The subgroup B of Perm Y which preserves the icosahedral structure 8 operates transitively on Y and is isomorphic to the full icosahedral group h.

Now conversely assume that X is any 12-element set and A c Perm X is a subgroup which is isomorphic to I and which operates transitively on X. Recall that there exists 2 conjugacy classes F, F' of elements of order 5 in A. One defines an A-invariant icosahedral structure 8 on X by declaring that {x, y} is an edge, for distinct x, y EX, if there exists 9 E F such that g. x = y. Replacing F by F' defines a second A-invariant icosahedral structure 8'. Then since any two transitive actions of I on a 12-element set are isomorphic and since an isotropy group of this action has index 2 in its normalizer, one readily proves

Proposition 1. There exists exactly 2 icosahedral structures 8,8' on X which are invariant under A. Furthermore the centralizer c = c(A) of A in Perm X is a group of order 2 and the orbits of c are the sets of antipodal pairs for both 8 and 8'. Also x, y E Yare neighbors for 8 if and only if they are coneighbors for 8' .

For any 12-element set X, let S be the set of all icosahedral structures on X so that S has the structure of a Perm X -set. For any 8 E S let As C Perm X be its (icosahedral) symmetry group. If 8 E S then a distinct 8' E S is called its twin, and 8, 8' are referred to as a pair in case As = As'.

We apply these considerations to the case where X = Pu, the projective

Groups and the Buckyball 105

line over a field Fll of H elements. We may regard Pl(2, H} C PermX so that Pl(2, H} is a 660-element group of permutations of X.

An icosahedral structure S on X will be said to be Pl(2, H} compatible if As C Pl(2, H}. We will be interested in such structures. The set C of 6-pair partitions of X, considered above, may clearly be identified with the set of all subgroups c C Perm X of order 2 such that c has no fixed points on X. As noted above there exists Co E C whose centralizer, Ao, in Pl(2, H} is isomorphic to I and which operates transitively on X. Obviously

c(Ao) = Co.

In particular Pl(2, H) contains a subgroup which is isomorphic to I. Let A be the Pl(2, H}-set (under conjugation) of all subgroups of Pl(2, H) which are isomorphic to I. If A E A and Zll is any cyclic subgroup of order H in Pl(2, H) then since obviously An Zl1 = {e} considerations of order immediately implies that

Pl(2, H) = AZll . (1)

By the simplicity of Pl(2, H), obviously then, any A E A is equal to its own normalizer in Pl(2, H). Any element in a E Gl(2, H) whose determinant is not a square induces, by conjugation, an outer automophism a of Pl(2, H) which interchanges the 2 conjugacy classes of elements of order H but does not interchange the 2 conjugacy classes of elements of order 5. This is clear by noting that this is obviously the case for diagonal elements in GL(2, H). It follows that if A E A then a(A) is not conjugate to A. Indeed otherwise there would exist a non-central element in b E GL(2, H) which commutes with A. This is clearly impossible since by conjugation (noting that any cyclic group of order 5 is a Sylow subgroup) we can assume D. C A, i.e. A contains all the image of the set of diagonal elements of determinant 1. But that forces b to be diagonal and non-central. Thus if AI is the Pl(2, H)-orbit in A which contains Ao and All = a(AI), then AI =I- All.

Proposition 2. There are 22 elements in A. Furthermore, under the action of Pl(2, H), A decomposes into two ll-element orbits

A = AI UAII

Mo'reover there exists a' E PermX, in the normalizer of Pl(2, H), which induces an outer automorphism a of Pl(2, H) interchanging AI and All. Correspondingly there exist 22 pairs of Pl(2, H) compatible iscoahedral struc­tures on X. Again correspondingly they fall into 2 Pl(2, H)-orbits.

Proof. We will show that if A E A and A ~ AI then A E All. Let A E A \ AI. Furthermore, as above, up to Pl(2, H)-conjugacy, we can

106 F. R. K. Chung, B. Kostant, and S. Sternberg

assume that Do c A. Choose the element a E Gl(2, 11) which induced a to be diagonal so that a fixes the elements of Do. Consider the action of Pl(2, 11) on the 11-element set Cu. Since A ¢ AI it has no fixed point on Cu. Thus the only possible orbit structure it can have on Cu is two orbits 0 5 , 0 6 with respectively 5 and 6 elements. On the other hand, as we have seen in the preceding section, the subgroup Do c A, has exactly two 5-element orbits, D = (Fid 2 and D' = -(Fi1?on Cu. Thus one must have 0 5 = D or 0 5 = D'.

As our subgroup Ao also contains Do and a preserves Do the group a(Ao) will also have a five element orbit on Cu which will be either D or D'. Let us denote its five element orbit byOg. Now suppose that A' = a(A) ¢. AI. Let O~ and O~ be the 5 and 6 element orbits of A' in Cu. Again O~ = D or D'. But since A' is not conjugate to A one cannot have 0 5 = o~. Indeed, by (1), any two distinct elements of A \ AI generate Pl(2, 11) and hence cannot have the same 5-orbit in Cu. Thus {05,OD = {D,D'}. But og E {D, D'}. Thus a(Ao) is equal to A or A'. This is a contradiction since a 2 is clearly an inner automorphism. Thus AI and AI I exhaust A. The group Gl(2, 11) operates on the projective line X. Clearly if one defines a' to be the permutation action of a then a' normalizes Pl(2, 11) and induces a on Pl(2, 11). QED

Let AI be normalized as in the proof of the preceding proposition so that Ao E AI. We now observe that Cu is not the only 11-element Pl(2, 11)­orbit in C.

Proposition 3. Let

C~l = {c(A)IA E All}

Then q1 is an ii-element orbit of Pl(2, 11) in C and C~l is distinct from Cu.

Proof. Obviously C~l = a'Cu o:'-l so that Cit is an 11-element Pl(2, 11)­orbit. It is distinct from Cu since as noted in the proof of Proposition 2, no element A E All has a fixed point on Cu , whereas c(A) is fixed by A. QED

Let A E AI. Associated to A is a pair of A-invariant icosahedral struc­tures 8,8' on the projective line X. The 2-element permutation group c(A) tells the pairs of antipodal points for 8 and 8' but it does not tell us what the edges are. Remarkably the cross-ratio [Xl. X2, X3, X4] for Xi E X can be used to determine at least one of the two structures 8,8'. Elsewhere we

Groups and the Buckyball 107

will prove

Theorem 1 Let T E c(A) be the non-trivial element. Let x, y E X with {x, 11., V, y} stable under T. Then {x, y} is an edge for an A-invariant icosa­hedral structure on X if and only if

[x,u,v,y] =-1.

4. Representations

The representations of all the alternating groups were determined by Frobe­nius: Regard An as a subgroup of 8n. The irreducibles of 8 n are given by Young diagrams. If a Young diagram is not the same as its conjugate di­agram, the representation remains irreducible when restricted to An and is equivalent to the restriction of the conjugate diagram. A self conjugate diagram splits into two irreducibles of half the dimension. For the case of 8s , the irreducibles can be labelled as 1,4,5,6,5' ,4',1 where the ' describes conjugate diagrams and the 6, corresponding to the partition [3,1,1], is self conjugate and so splits into two three dimensional representations, so we can list the representations of I as 1,3,4,5,3'. They can be described as follows:

The action of Pl(2,p) on P! is doubly transitive, and so the induced ac­tion on the p-dimensional space of functions, .r(P!) splits into two pieces, the constants and an irreducible of dimension p called the Steinberg repre­sentation. For the case p = 5, this gives the five dimensional representation. The four dimensional representation is the restriction of the fundamental representation of 8 s: the group As acts doubly transitively on the set of five objects and so the permutation representation splits into the constants and a four dimensional irreducible. Equally well, we can regard 4 as the Steinberg representation of 81(2,4). We will have occasion to use the case p = 11 of the Steinberg representation in section 7. The two 3-dimensional representations are two inequivalent ways of regarding As as a subgroup of 80(3) depending on whether the five cycle (12345) is sent into a rotation through angle 27r /5 or 47r /5.

The conjugacy classes of As are easy to determine from those of 8s. If

p = (ab ... ) (c .. ) ..

is a permutation written in cycle form, and 7r is some other permutation, then

7rP7r- 1 = (7r(a)7r(b)· ··)(7r(c)··)· ..

108 F. R. K. Chung, B. Kostant, and S. Sternberg

So a conjugacy class in 8n is determined by its cycle structure. The conjugacy classes of even permutations in 85 are

e (a b) (cd) (abc) (abcde).

It is easy to check that any two (ab)(cd) are conjugate to one another by an even permutation as are any two (abc). But the element s = (12345) is not conjugate to its square s2 = (13524) by any even permutation, since any conjugating permutation is determined up to right multiplication by a power of s (which is even) and one such conjugating permutation is the four cycle (2354) which is odd. So the five cycles split into two conjugacy classes in A5 • Thus the table of conjugacy classes is as follows:

size order rep

1 15 1 2 e (12)(34)

20 3

(123)

12 5

(12345)

12 5

(13524)

In a permutation representation, the character of any element is equal to the number, fix(x), of fixed points. Hence, when we subtract off the constants, the character, x(a), of the remaining representation for any ele­ment a is given by fix (a) - 1. For the Steinberg representation, the order two elements have two fixed points, the order three elements none, and the order five elements one fixed point. (These facts can be checked on the axes of the buckyball, but will become even more transparent in terms of the representatives in 8l(2, 5) that we will choose later on.) Thus, in terms of the order of conjugacy classes in the above table, the character of the five dimensional irreducible has values 5,1,-1,0,0. (Check: 25+15+20 =60.)

Acting as permutations on five letters, order two elements have one fixed point, order three elements have two, and order five elements have none. So the character values for the four dimensional representation are 4,0,1,-1,-1. (Check 16+20+12+12 =60.)

The trace of any rotation through angle () in three dimensional space is 1 + 2 cos () = 1 + eiO + e-iO • In all that follows we will set

If we choose our identification of A5 with the icosahedral group so that (12345) goes over into rotation through angle 27l' /5, the character values are 3, -1, 1, 1 + (+ (4, 1 + (2 + (3. The other three dimensional representation will then correspond to the opposite choice so that (12345) corresponds to rotation through 47l'i/5. This will have the effect of interchanging the last

Groups and the Buckyball 109

two values. So the character table for A5 is given by

size 1 15 20 12 12 order 1 2 3 5 5 rep e (12)(34) (123) (12345) (13524) trivial 1 1 1 1 1 3 3 -1 0 1+10+104 1 + 102 + 103

5 5 1 -1 0 0 4 4 0 1 -1 -1 3' 3 -1 0 1 + 102 + 103 1+10+104

For the purposes of applying Frobenius reciprocity later on, it is impor­tant to record how each of these representations restricts to a two element subgroup of I consisting of the identity and an element of order two, that is, how many times the sign representation occurs and how many times the trivial representation occurs in the restriction. A rotation through 1800 in three dimensional space has two eigenvalues -1 and one eigenvalue + 1. So the restriction of each of the three to Z2 contains the sign representation twice and the trivial representation once. The restriction of the permutation representation of A5 acting on five objects to Z2 is clearly the direct sum of the trivial representation and two copies of the two dimensional permu­tation representation. Hence the four dimensional representation restricted to Z2 contains two copies of the trivial representation and two copies of the sign representation. For the five dimensional representation we record the following fact which has applications to the Raman spectrum.

Proposition 4. The representation of I on S2(R3) decomposes into 1 E9 5 where the 1 corresponds to multiples of the Euclidean metric and the 5 consists of the tmceless symmetric tensors.

Proof. The decomposition into multiples of x2 + y2 + z2 and traceless tensors is clearly invariant under I. On the other hand, there can not be any other invariant line in S2(R3) since this would contradict the irreducibility of the three dimensional representation. Since there is no two dimensional irreducible representation, we see that there is no way for the the traceless tensors to decompose. QED.

If we choose our rotation to be about the z-axis, the monomials z2, x2, y2, xy are invariant while xz and yz are eigenvectors with eigenvalue -1. Hence the 5 when restricted contains three copies of the trivial representation and

110 F. R. K. Chung, B. Kostant, and S. Sternberg

two copies of the sign representation. To summarize we have the table

rep 1 3 4 5 3'

tr 1 1 sgn 0 2

2 3 2 2

1 2

(2)

The group Ih is just the direct product of I with Z2, so the irreducibles of Ih are just the tensor product of those of I with those of Z2' In other words, they can be labeled by attaching a + or - sign to the label of the representation of I according to whether it is the trivial or sign representa­tion of Z2 which is tensored in. (In the chemical literature this ± labeling is given by a subscript g (for gerade) for the + and a subscript u (for umger­ade) for the -.) The subgroup of Ih which fixes a point, call it H, is a two element group consisting of the identity and Pr where r is rotation through 1800 and P is the parity operator. So the table corresponding to (2) where now we list the restrictions to H is

rep

tr 1 sgn 0

1 2

2 2

3 2

1 2

o 1

2 1

2 2

2 3

2 1

(3)

By Frobenius reciprocity we can read the entries of this table as giving the multiplicities of each irreducible of h in the representation induced from the trivial or the sign representation of H. In the electronic spectrum one is particularly interested in the representation induced from the sign representation, hence the second row of (3). In the vibrational spectrum one is interested in the representation induced from the restriction of 3-to H. From the above table the restriction of 3- is given by

3- != 2· tr EB 1 . sgn

and hence the corresponding induced representation, (3- !) 1 is given by

Thus finding the eigenvalues of any h invariant operator on this space involves dealing with at most an eight by eight matrix, as was pointed out in [6]. The vibrational spectrum under the assumption of bond stretching and angle bending forces was calculated in [6]. The space of vibrational states is the above induced representation with one 3- (overall translations) and one 3+ (overall rotations) removed. The infrared lines corresponding to vacuum to one phonon (dipole) transitions correspond to these four representations,

Groups and the Buckyball 111

and hence, by (4) there are four lines in the infrared spectrum. The Raman spectrum corresponds to a quadrupole interaction, and hence to symmetric tensors of order two. By Proposition 1 and (4) we see that there should be ten Raman lines in all, two corresponding to 1+ and eight corresponding to 5+, and all ten lines have been observed. See [2] for a picture of these ten lines.

Let us now turn to the double cover, and begin by describing the con­jugacy classes, which are best described in terms of familiar subgroups. We have already described the Borel subgroup B consisting of all upper triangular matrices. We can write B = TU where T is the subgroup of diagonal matrices and U consists of upper triangular matrices with ones on the diagonal. (The subgroup T is called a split torus and the elements of U are unipotents.) In general, T will contain q - 1 elements, in our case of q = 5 this comes to 4. One of these elements is id, another is -id and the remaining two elements are of order four. They are conjugate to one another, say by the matrix

So the fifteen order two elements in As are each covered by two elements of order four in 8l(2, 5) giving thirty elements in all in this conjugacy class.

The matrix M = (~ ~) is of order five and so must project onto one

of the elements of order five in As. But -M is of order ten and projects onto the same element. So we see that the inverse image of each of the two twelve element conjugacy classes in As splits into two conjugacy classes, one consisting of elements of order 5 and the other of order ten. The

matrix N = ( 0 1) is of order three, and hence maps onto an element of -1-1 order three, and -N is of order six and maps onto the same element. So the inverse image of the elements of order three in As also splits into two conjugacy classes, one consisting of order three elements and the other of order six elements. So there are nine conjugacy classes in all, as listed in the following table:

size order

rep

1 1 1 1

1 -1

30 20 20 12 12 12 12 4 6 3 10 10 5 5

C~) (-~~) (-~-~) (-~=~) (-~=~) (~~) CD A cyclic subgroup generated by an element of order six has the following

interpretation in terms of the general theory of 81(2, q): The field F q has a

112 F. R. K. Chung, B. Kostant, and S. Sternberg

unique quadratic extension, F q2 which we may regard as a two dimensional vector space over F q' The set of non-zero elements in F q2 form a group under multiplication and multiplication by an element of F q2 gives a linear transformation of F~. There are q2 - 1 non-zero elements in F q2 and q - 1 values of the determinant, and so the elements whose multiplication have determinant 1 form a subgroup of order q + 1. If we chose a basis of F q2 we get subgroup, K of 8l(2,q) of order q+l called a 'non-split torus'. It is the finite field analogue of the unit circle in the extension of the real numbers by the complexes. In our case q + 1 = 6 and each non-split torus covers a C3 subgroup of A5 generated by a three cycle.

In any event, since there are nine conjugacy classes, there are nine in­equivalent irreducible representations, and we have found five of them, the representations coming from those of A5 • The general theory of the rep­resentations of 8l(2, q) has been completely worked out, cf. for example [11). Assuming the existence of a two dimensional representation, which we have already proved, we can work out the character table for our special case of 8l(2, 5) as follows. We identify our group with the inverse image of A5 under the projection of 8U(2) onto 80(3). Now the group 8U(2) has one irreducible representation of every dimension, the representation k, of dimension k being on the space of homogeneous polynomials of degree k - 1 in two variables. The value of the character of the representation k on a matrix whose eigenvalues are eiO , e- iO is

Xk(B) = e-(k-l)iO + e-(k-3)iO + ... + e(k-l)iO.

In particular, for k = 2 we have

So the character of the two dimensional representation is given by the second line in the table below. The first six lines of the table are just the evaluation of the characters Xk on our 120 element subgroup of 8U(2) for k = 1,2,3,4,5,6. A check shows that the sum of the absolute value squared of the elements multiplied by the number of elements in the conjugacy class adds up to 120 and shows that these characters are indeed irreducible. Thus k remains irreducible when restricted to our subgroup up until k = 7, which splits into a 4 and a 3. Here is the character table.

Groups and the Buckyball 113

size 1 1 30 20 20 12 12 12 12 arder 1 2 4 6 3 10 10 5 5

/ -/ (~~) C~~) C~-~) ( -1-1) ( -1-2) (~~) G~) rep 0-1 0-1

tr 1 1 1 1 1 1 1 1 1 2 2 -2 0 1 -1 _«2+<3) _«+<4) f'2+E3 <+<4

3 3 3 -1 0 0 <+<4+1 <2+<3+1 <+<4+1 <2+<3+1

4 4 -4 0 -1 1 1 1 -1 -1 5 5 5 1 -1 -1 0 0 0 0 6 6 -6 0 0 0 -1 -1 1 1 4' 4 4 0 1 1 -1 -1 -1 -1 3' 3 3 -1 0 0 <2+<3+1 <+<4+ 1 <2+<3+1 <+<4+1

2' 2 -2 0 1 -1 _«+<4) _«2+<3) <+<4 €2+E'3

5. The Clebsch Gordan decomposition for Sl(2,5)

For many applications, it is important to know the decomposition of the tensor product of two irreducibles. For the case of SU(2), the formula above for Xk shows that

X2· Xk = Xk-1 + Xk+1

which is, of course a special case,

2 ® k = (k - 1) EEl (k + 1),

of the Clebsch Gordan decomposition. So we can apply this to each line in our table, multiplying by the second line and subtracting the preceding line to get the next line up until line 6. There, multiplying the 6th line by the 2nd and subtracting the fifth gives the sum of the next two lines. Also multiplying the 4 by the second line gives the sum of the 6 and the (other) 2 and multiplying the 3 by the second line gives the 6. We can summarize this discussion by saying that the effect of tensoring by 2 in the basis of irreducibles is given by the adjacency matrix of the graph

1 2 3 4 5 6 4' 2' • • • • • I • •

3'

114 F. R. K. Chung, B. Kostant, and S. Sternberg

which is the extended Dynkin diagram of E8 • The integers 1,2,3 etc. placed near the nodes, i.e. the dimensions of the irreducible representations of G are the coefficients of the highest root of E8! This is a case of the McKay correspondence which similarly associates a simple Lie algebra to every finite subgroup of SU(2).

Using the above diagram recursively, or by examining the character table and seeing that the character of 2 separates all conjugacy classes, we conclude that every representation can be written as a polynomial in the basic two dimensional representation, 2. Explicitly the representation k = !k(2) where the polynomials fk are given by

h(x) 1

h(x) x

h(x) x 2 -1

f4(X) x 3 -2x

f5(X) = x4 - 3x2 + 1

f6(X) x5 - 4x 3 + 3x

f4'(X) = x8 _ 7x 6 + 14x4 - 7x2

!Jt(x) _x8 + 8x6 - 19x4 + 13x2 - 1

h(x) = x 7 - 7x5 + 14x3 - 7x.

Thus the Grothendieck ring of G is given by Z[xl/p(x) where p(x) is the characteristic polynomial of A, where A is the adjacency matrix of the extended Dynkin diagram. This can be verifed directly, or, from the fact that

2 ® 4' = 6 EI1 2'

(which can be verified from the character table). If we substitute the polynomial expressions given in the preceding table for both sides of the above equation, the difference between both sides will give the characteristic polyno mial of A.

By using a general theorem of Kostant [7), tensoring by any representation is given by multiplication by the corresponding polynomial listed above, with the understanding that two polynomials are to be identified if they differ by a multiple of the characteristic polynomial.

In the study of the electronic properties of C60 one also needs to know the decomposition of the exterior powers of various representations, as these contain the Fermi allowed states. We discuss this problem in the next two

Groups and the Buckyball 115

sections. In the next section we discuss the general method, and in the following section we use some tricks special to the group G.

6. Decomposing exterior powers

There is a general method which always works. In any given case, it might be simpler to use some special tricks. The general method is as follows: For any character, X, let Pk(X) denote the function on conjugacy classes,

If p denotes the representation and the eigenvalues of p(a) are Xl, ••• ,xn

then x(a) = LXi

while Pk(x)(a) = LX:.

On the the other hand, the eigenvalues of the k-th exterior power of p(a) are

XilXi2·· ·Xik' il < i2 < ... < ik

so the character of the exterior power is

The problem is how to express the E's in terms of the piS. For example,

and

E2 = ~ [Pf - P2 ] = ~ det (~l ~~). The character on the second exterior power is

Similarly,

Thus

116 F. R. K. Chu.ng, B. Kostant, and S. Sternberg

The general formula expressing the E's in terms of the p's is

PI P2 P3 Pk- I Pk 1 PI P2 Pk-2 Pk-I

1 0 2 PI P2 ··· Pk-2 Ek = k! det

0 0 0 k-l PI

In principle one can use this formula to compute the characters of the exterior power from the character table. In practice, this computation might get complicated.

7. Special tricks

Let U and V be vector spaces. Then

/\k (U EB V) = EB /\i(U) ® /\j(V). i+j=k

(5)

A similar formula works for the exterior power of the direct sum of any number of vector spaces. In particular, the calculation of the exterior power of any representation is reduced to the calculation for irreducibles.

If a representation is self contragredient (as is the case for all the irre­ducible representations of G = 8[(2,5» then /\n-k ~ /\k. So we need only compute the /\2 of all the irreducible representations and also the /\3 of the six dimensional representation. For the computation of the /\2 , we will use (5) and the well known fact

(6)

We can use (5) and (6) to recursively determine all the /\2,S, starting with

/\22 = tr = 1

and /\23 = 3.

From the multiplication table we get

Then using (6)

/\2(2 ® 3) = 1 EB 5 EB 3 ® 3 = 1 EB 5 EB 1 EB 3 EB 5.

Groups and the Buckyball 117

But 2 ® 3 = 4 EB 2 and from (5)

,,2(4 EB 2) = ,,2(4) EB 4 ® 2 EB ,,2(2) = ,,24 EB 3 EB 5 EB 1.

From these two equations and the formula for 4®4 from the multiplication table we conclude that

Proceeding in this way we find

,,22 = 1 ,,23 3 ,,24 = 1EB5 ,,25 3 EB 3' EB 4' ,,26 = 1 EB 2 x 5 EB 4'

,,23' = 3' ,,24' = 3 EB 3' ,,22' 1

To compute ,,3(6) we use a theorem, see e.g. [7] Corollary 8.2, which generalizes (6). It says that the decomposition of "k(X ® Y) into irre­ducibles under Gl(X)xGl(Y) is given as follows, when dimX = m, dim Y = n: Construct all Young diagrams with k boxes which fit into a rectangle of size m x n. Any such diagram (with at most m rows and n columns) corresponds to an irreducible representation of Gl(X). The same diagram flipped over (so it now has at most n rows and m columns) corresponds to an irreducible representation of Gl(Y). So if we take the tensor prod­uct of these two representations we get an irreducible representation of Gl(X) x Gl(Y). A special case of Corollary 8.2 in [7] says that "k(X ® Y) decomposes into the direct sum of the representations associated as above with all possible diagrams with k boxes which fit into the rectangle of size m x n. In case m = 2, n = 3, the only possible diagrams are [3] and [2,1]. The corresponding representations are 83(X) ® ,,3(y) and X ® U where U is an eight dimensional representation of Gl(Y). This eight dimensional representation occurs twice in the decomposition of Y ® Y ® Y. Explicitly

We wish to apply this to the case X = 2, Y = 3'and write 6 = 2®3'. We must now decompose these representations when restricted to the double cover of I. Now ,,33' = 1,832 = 4 so the representation corresponding to [3] remains irreducible and becomes 4. To compute the term corresponding

118 F. R. K. Chung, B. Kostant, and S. Sternberg

to [2,1] we must know how U decomposes. For this we check from the multiplication table that the only irreducibles which occur with multiplicity ~ 2 in 3' ® 3' ® 3' are 3' and 5. so

U = 5 EI7 3'.

From the multiplication table,

2 ® [5 EI7 3'] = 4 + 2 x 6.

We conclude

8. The electronic spectrum and the Galois embedding

In the Hiickel model, the electronic energy levels are determined by the spectrum of the adjacency matrix of the graph of the buckyball. The ad­jacency matrix acts on the space of functions on the vertices, and depends (up to scale) on one real parameter, t, the relative strength of the dou­ble bonds to the single bonds. According to (3), finding the spectrum of any invariant operator on the space of functions involves diagonalizing at most a three by three matrix, and hence can be solved in closed form. In general, the eigenvalues for the adjacency matrix of a homogeneous graph can be treated by the method of Frobenius, cf. [2] for a modern exposition and [3] where this is worked out for the Buckyball and the characteristic polynomials as a function of t were determined to be

(a) (x2 + x - t2 + t -1)(x3 - tx2 - x 2 - t 2x + 2tx - 3x + t3 - t 2 + t + 2) = 0 with multiplicity 5;

(b) (x2 + x - t 2 - 1)(x2 + X - (t + 1)2) = 0 with multiplicity 4;

(c) (x2 + (2t + 1)x + t2 +t _1)(X4 - 3x3 + (_2t2 +t -1)x2 + (3t2 - 4t + 8)x + t4 - t3 + t2 + 4t - 4) = 0 with multiplicity 3;

(d) x - t - 2 = 0 with multiplicity 1.

Groups and the Buckyball 119

If we draw the eigenvalues as functions of t, we obtain

0.5 1.5 2 2.5 3

confirming by closed analytic expression the computational results of [5]. The fact that at the limit t = 0 we get the eigenvalues 2,2 cos 271"/5 = f. + f.4,

2 cos 471" /5 = f.2 + f.3 is not surprising, because at t = 0, all the pentagons disengage, and these are the eigenvalues of the cyclic pentagonal graph. What is more interesting is the decomposition into irreducibles of I or h when t is perturbed away from o.

To understand this, recall that the eigenfunctions on the cyclic graph are exactly the characters of the cyclic group, the character taking the value w on the cyclic generator having eigenvalue 2 cos w. Thus the space of functions on the buckyball with eigenvalue 2 can be identified with the space of functions on the set of pentagons, i.e. on Pb, the eigenfunctions with eigenvalue 2 cos 271" /5 correspond to sections of the line bundle on P~l induced from the characters which take the values f. and f.4 on the generator, and the eigenfunctions with eigenvalue 2 cos 471" /5 correspond to sections of the line bundle on P~l induced from the characters f.2 and f.3. If we examine the restriction for the various representations of I to Z5 we have the following table:

rep 11 rep! 1

3 4 5 1,f.,f.4 f.,f.2 ,f.3 ,f.4 1,f.,f.2 ,f.3 ,f.4

Thus the break up into irreducibles when t is perturbed away from 0 is

120 F. R. K. Chung, B. Kostant, and S. Sternberg

just Frobenius reciprocity. Recall the embedding of Sl(2, 5) into Sl(2, 11) which embeds the subgroup fixing a point as the diagonal matrices. From the point of view of Sl(2, 11), the space offunctions on pll is just the direct sum of the trivial representation and the Steinberg, while the representa­tions induced from € (isomorphic to that induced from €4) is a principal series representation as is the representation induced from €2 which is iso­morphic to that induced from €3. So the above graph sees the restriction of these representations to I.

There is an interesting remark about the restriction of the Steinberg: The restriction of the irreducible representations of SU(2) to a finite sub­group were all worked out in[8]. We will summarize the result for the case of Sl(2, 5) in the next section. In particular, the spin 5 eleven dimensional representation breaks up upon restriction to G as 5 EB 3 EB 3', the same as the restriction of the eleven dimensional Steinberg. When we pointed this out to Dick Gross, he observed that the same is true for all the exceptional Pl(2,p) which have a transitive permutation action on p letters. In fact, following a letter from Gross, let G be one of these exceptional Pl(2,p) acting on p letters, and let K be the stabilizer of a letter so (G : K) = p. We have the table

p G K 2 Pl(2,2) = S3 Z3 3 Pl(2, 3) = A4 C2 X C2

5 Pl(2, 5) = A5 A4 7 Pl(2,7) = Sl(2, 3) S4 11 Pl(2,11) A5

Note that K can always be realized as a subgroup of SO(3), and when p = 2, K lifts to a subgroup of SU(2). Let V be the Steinberg representation of G, so dim V = p, and let W be the unique irreducible representation of SU(2) of dimension p. When p > 2 is odd, W descends to a representation of SO(3). The observation is that in all these cases, the restrictions WIK and YjK are isomorphic as K modules and are multiplicity free. In fact we have the following table of decomposition:

P YjK r"V W1K 2 1 EB I' 3 1 EB I' EB I" 5 1 EB I' EB 3 7 1 EB 3 EB 3'

11 3 EB 3' EB 5.

Groups and the Buckyball 121

9. Homogeneous magnetic energy

In a fundamental paper of 1937 [10], London gave a generalization of the Huckel model to account for the presence of a magnetic field. It involves generalizing the notion of the adjacency matrix of a graph: Let:F denote the space of functions on the set of vertices of our graph. If we have assigned real numbers (called bond strengths) to each of the edges of a graph, then the adjacency matrix A is the operator

A::F --+:F, (Af)(u) = L aef(v)

where the sum is over all vertices v adjacent to u, where e denotes the edge joining u to v, and where ae is the weight attached to the edge e. Suppose we consider a (Hermitian) line bundle, L over the vertices of the vertices of the graph, and replace :F by the space r(L) of sections of L. Then the above definition does not make sense, since we have no way of comparing Lv with Lu. To get an adjacency matrix, we must also specify a connection, 9. This means that for every vertex, x, and every edge, e emanating from x we must be given a unitary map

where y is the other end of the edge e. The sole condition is that

Tx,e 0 Ty,e = id

for all edges e where x and y denote its end points. There is no hypothesis about the composition of the T's along edges of a path. In particular, the composition of the T's along the edges of a closed path starting and ending at x will be given by multiplication by a complex number of absolute value 1, known as the holonomyor the flux of the connection, 9 around the path.

Now, for each choice of connection (and bond strengths), define

A(9) : r(L) --+ r(L), A(9)f(u) = L aeTv,ef(v).

In matrix terms, we might think of the original definition of the adja­cency matrix as referring to a choice of trivialization of the line bundle, with corresponding trivial connection. Then a choice of e amounts to a choice of phase factor ei8"'11 for each non-zero entry of A, with

122 F. R. K. Chung, B. K ostant, and S. Sternberg

and replacing

The operator A(0) is self adjoint and has trace zero. It is easy to check that the spectrum of A(0) depends only on the holonomy of 8. London's generalization of the Huckel model says that the presence of a magnetic field gives rise to a choice of connection. As trA(8) = 0, an important invariant is

EA(8) = L A(A(0)) = ~ L IA(A(0))1, A>O

the sum of the positive eigenvalues. As in the Huckel model, the larger EA(9), the lower the ground state energy for the n-electron state (where n is the number of vertices, assumed even for simplicity). London uses this theory in [10] to discuss the diamagnetic properties of aromatic compounds.

In a recent paper,[9] Lieb and Loss discuss the following problem: For a given choice of bond strengths, A, find the connection (or rather the holon­omy) which maximizes EA(0). They show that for some special classes of planar graphs, such as cycles, necklaces, and some "tree-like" graphs, the maximum is achieved at the canonical flux one which associates flux 11"/2 to each oriented triangle in a triangulation of the graph. (One can show that if such a flux exists, it is independent of the triangulation; and that the canonical flux exists for the class of graphs they consider.) What is re­markable about their result is that the answer is independent of the choice of bond strengths. This is a property of the special class of graphs they consider.

For the case of the buckyball, a general magnetic field will destroy icosa­hedral symmetry. But we can ask the Lieb-Loss question for homogenous connections, i.e ones which are G- invariant: For a fixed value of the double bond strength, t, which G-invariant connections maximize Et(9)?

In order to deal with this question, we must first list the G-equivariant line bundles. Since the isotropy group of a vertex is H = Z2, the trivial bundle and the non-trivial bundle correponding to the sign representation of Z2. If we denote the non-trivial bundle by L, then F'robenius reciprocity says that r(L) decomposes as

r( L) = 2 x 2 E9 4 x 4 E9 6 x 6 E9 2 X 2',

i.e. as a direct sum of the representations of G which don't descend to I, each occurring with a multiplicity equal to its dimension.

For a general homogeneous graph, with H the isotropy group of a cho­sen, vertex, v, and BeG the set group elements which move v to an

Groups and the Buckyball 123

adjacent vertex, specifying a G-invariant connection on a G-equivariant line bundle amounts to assigning a complex number Zb with IZbl = 1 to each b E B subject to the conditions

Zhb = Zbh = p(h)Zb, hE H, bE B, and Zb-1 = z;;1

where p is the character determining the line bundle, cf. [1]. In the case at hand, this last condition implies that Zw = ±1 for the double bond generator in the case of the trivial bundle, and Zw = ±i for the double bond in the case of the non-trivial bundle, but there is an arbitrary choice of phase for a pentagonal generator. so in each case, the space of G-invariant connections consists of a circle. Here in the figure (top of following page) , we illustrate the values of EI ((J) where (J ranging from 0 to 211" for the case of t = 1. Notice that the absolute maximum is achieved for the non-trivial bundle and with canonical flux! In particular, this gives a minimum energy justification for passing to the double cover. (Had we taken t = 0 and hence decoupled into twelve pentagons, the maximum would be achieved at phase 0, so the result does depend on the bond strengths.)

10. Restriction from 8U(2)

For various reasons it is important to know how the irreducible represen­tations of 8U(2) restrict to the subgroup G = 8l(2,5). This problem was completely solved in [8] for all finite subgroups of 8U(2). We record the results for 8l(2, 5) here. If we let ak,r denote the multiplicity of the repre­sentation T in the restriction of the spin k/2 representation of 8U(2) to G, then we can form the generating series

We list these series for each of the nine values of T. Each generating series is a rational function, and they all have the same denominator: we have

where only the numerator, the polynomial Pr depends on the representa­tion, T. SO we must list the polynomials Pro They are given as

PI = 1 + t30

P2 t + ttl + t I9 + t 29

P3 t 2 + t lO + t I2 + tIS + t 20 + t 2S

P4 t 3 + t 9 + ttl + t I3 + t I7 + t I9 + e1 + t 27

124 F. R. K. Chung, B. Kostant, and S. Sternberg

= t4 + t8 + t lO + t l2 + t l4 + t l6 + t l8 + t20 + t26

t5 + t1 + t9 + tll + t l3 + 2tl5 + t l7 + t l9 + t2l + t23 + t25

t 6 + t 8 + t l2 + tl4 + t l6 + t lS + t22 + t24

= t 6 + t lO + t l4 + t l6 + t 20 + t24 = t 7 +tl3 +tl7 +t23

where 4'" denotes the representation of G which descends to the four di­mensional representation of I. In particular, note the highly suggestive form for the generating function of the invariants:

1 + t 30

Zl (t) = (1 _ tl2)(1 _ t20)

where 30 is the number of edges, 12 the number of vertices, and 20 the number of faces of the icosahedron .

.j 7

2lt

Groups and the Buckyball 125

The integer spin (corresponding to even values of k) representations of 8U(2) descend to representations of 80(3), and can be realized as the action of 80(3) on harmonic polynomials on R3 of degree ~k. The full group 0(3) acts on these spaces, with P acting as +id if ~k is even and as --id if ~ k is odd. So for even values of k we can use the above series to determine the restriction of the harmonic polynomial representation to h. For example, from the above series for Zl we see that the first few occurrences of the trivial representation are for the k values k = 1,12,20,30 and of these k = 30 is the first case where ~ k is odd. So the first occurrence of 1- is in spin fifteen.

11. The covers of 0(3)

The connected group 80(3) has the unique double cover, 8U(2) which is its universal cover. But this is not the case for the disconnected group 0(3). There are basically two natural choices: if Q denotes one of the two elements projecting onto the parity operator, P, we can have Q2 = id or Q2 = -id. In other words Q can be either of order two or of order four. The "covering group" can be either 8U(2) x Z2 or U(2). (In terms of Clifford algebras and Pin groups, the element in the Pin group covering P will be Q = ele2e3 where the e's form an orthonormal basis. So the question of whether Q is of order two or four hinges on whether the metric we are using is negative or positive definite. In terms of representation theory this difference expresses itself as follows: In both cases all representations of the double cover carry an additional label of ± along with the spin. For even spin, in both cases the ± means that Q is represented by ±Identity. But for odd spin, if the group is 8U(2) x Z2 the ± means that Q is represented by ±Identity, while if the group is U(2), then the ± signifies that the chosen generator Q is represented by the scalar ±i. Since (±1)3 = ±1 while (±i)3 = =Fi this makes a difference in terms of selection rules. Although these two possibilities were recognized as early as 1937 by Racah, they are not usually mentioned in the physics literature.

It is, of course, a matter of physics to determine which of the two choices describes the real world. From the point of view of the group G, the choice of U(2) as covering group is more pleasant, since, for this choice, the 240 element group covering h becomes a complex reflection group. Indeed, the elements in G which project onto rotations through 1800 are of order four - their square is -id E 8U(2). If we multiply them by the scalar matrix i, we get elements of order two which generate the covering group. Thus the covering group is a complex reflection group.

126 F. R. K. Chung, B. Kostant, and S. Sternberg

References

[1] Chung, F.R.K. and Sternberg, S. Laplacian and Vibrational Spectra for Homogeneous Graphs J. of Graph Theory 16 (1992), 605-627

[2] Chung F.R.K. and Sternberg, S. Mathematics and the Buckyball American Scientist 81 (1993) 56-71

[3] Chung. F.R.K., Rockmore, D. and Sternberg S. , Unpublished

[4] Conway, J. and Sloane, N. J. A. Sphere Packing, Lattices and Groups, Grundlehren der Mathematischen Wissenschaften 290, 1988 Springer­Verlag.

[5] Elser, V. and Haddon R.C. Magnetic behavior of icosahedral C60 Phys. Rev. A36 (1987), 4578-4584

[6] Weeks, D. E and Harter, W.G. "Rotation-vibration spectra of icosa­hedral molecules II. J. Chem Phys 90(1989)4744- 4771

[7] Kostant, B. Lie Algebra Cohomology and the Generalized Borel-Wei! Theorem, Ann. Math. (1961), 329-387.

[8] Kostant, B. The McKay correspondence, the Coxeter element and rep­resentation theory, Asterisque, 1985, Numero Hors Serie, 209-255.

[9] Lieb, E. H. and Loss, M. Fluxes, Lplacians and Kasteleyn's theorem, Duke Mathematical Journal (1993) 337-363

[10] London, F. TMorie quantique des Courants interatomiques dans les combinaisons aromatiques, J. Phys. Rad. 8 (1937) 397-409

[11] Naimark, M. A. and Stern, A. I. Theory of Group Representations, Springer-Verlag, New York 1982.

Fan R.K. Chung, Bellcore, 445 South Street, Morristown, NJ 07960

Bertram Kostant, Department of Mathematics, MIT, Cambridge, MA 02139

Shlomo Sternberg, Department of Mathematics, Harvard, Cambridge, MA 02138

Received December 17, 1993

Spinor and Oscillator Representations of Quantum Groups

Jintai Ding and Igor B. Frenkel

Dedicated to Bertram Kostant

I. Introduction

The theory of quantum groups originates in completely integrable models of statistical mechanics and quantum field theory (see [FRT], [J4]). It becomes a truly mathematical theory with the independent discovery by Drinfeld [Dl] and Jimbo [JI] of a q-deformation of the universal enveloping algebra of an arbitrary Kac-Moody algebra. This remarkable result immediately raised numerous questions about q-deformations of various structures asso­ciated to Kac-Moody algebras. A major step in this direction was done by Lusztig [L], who obtained a q-deformation of the category of highest weight representations of Kac-Moody algebras for generic or formal parameter q.

Before and after the work of Lusztig, there were a number of successful results on q-deformation of various mathematical structures of finite di­mensional and affine Lie algebras [D2], [J2], [FJ] , [H], [FR], etc. However each particular problem required its own insight combined with good luck and in some cases presented formidable difficulties. The main complication with the existing theory is the absence of the general invariant definition of the two most interesting subclasses of quantum groups associated to finite­dimensional and affine Lie algebras. The only available general definition using generators and relations obscures the invariant algebraic and geomet­ric nature of quantum groups and impedes some important applications. The necessity of an invariant approach to quantum groups was stressed in [FRT] , where a q-analogue of the matrix realization of classical Lie alge­bras was given. But even the latter definition is based on a remarkable but rather ad hoc solution of Yang-Baxter equation and can be viewed as the best available compromise.

In this paper, we would like to develop an invariant approach to var­ious representation theoretical constructions related to quantum groups. More specifically, we define quantum Clifford and Weyl algebras using the general representation theory of quantum groups. We show that the ex­plicit formulas for quantum Clifford and Weyl algebras match the ones

128 J. Ding and I. B. Frenkel

actively studied in the physics literature (see e.g. [PW] , [WZJ, [K]). Using those quantum algebras, we construct spinor and oscillator representations of quantum groups of classical types. Some of the formulas that follow from our construction including the algebraic manipulations with quantum Clifford and Weyl algebras related to the spinor and oscillator realizations of the quantum group Uq(g(n)) were also discovered in the physics lit­erature (see e.g. [IP]). Uniqueness arguments from representation theory allow us to justify that the quadratic expressions in quantum Clifford and Weyl algebras provide the desired representations. An explicit verification of Serre's relations for quantum groups leads to rather involved formulas (cf. [H]). In our approach, we avoid most of the tedious calculations by general arguments of representation theory. The key idea consists of re­formulating familiar classical constructions entirely in terms of the tensor category of highest weight representations and then, using Lusztig's re­sult on q-deformation of this category, defining the corresponding quantum structures. In the quantum case, we often need a quasi-triangular structure of the tensor category introduced by Drinfeld [D3], since it plays the role of the symmetric structure in the classical case. In particular, we would like to emphasize the central role of the universal Casimir operator implied by the quasi-triangular structure.

We can recover all the relevant formulas of quantum constructions after the main part of the work is completed. In order to formulate the quan­tum analogue of spinor and oscillator representations in invariant form, we first explain the representation theoretical origins of the well known classical counterparts. The classical types of finite-dimensional Lie alge­bras An, Bn, Cn, Dn and Lie superalgebras BCn admit particularly simple realizations as algebras of matrices with certain extra conditions. From the abstract point of view, it just manifests that a Lie algebra 9 of one of these types can be presented using the smallest fundamental representation V. Besides, the classical Lie algebras and superalgebras possess another distinguished representations known as spinor (for types An, Bn, Dn) and oscillator (for types An, BCn, Cn) representations. Again these representa­tions can be defined abstractly as irreducible representations with certain special highest weights. Moreover the Clifford and Weyl algebras, which play the key role in the constructions of representations, admit a descrip­tion in terms of the tensor category of representations. We will start from the Lie algebras of types Dn and Cn.

Let ~ be a Cartan subalgebra of a Lie algebra g. We choose a stan­dard orthonormal basis {Ei}f=l in ~* as in [VOl so that the root system Ll = {±Ei ± Ej}i,;=l' where i =1= j for Dn and i = j is allowed for Cn for equal signs. Let V be a 2n-dimensional irreducible representation of either

Spinor and Oscillator Representations of Quantum Groups 129

algebra with the highest weight El and let

V ® V = S(V ® V) E9 A{V ® V) (1.1)

be a decomposition into symmetric and antisymmetric parts. Then for the type D n , we have 9 ~ A(V ® V), while S(V ® V) contains a trivial representation which defines a symmetric from (,) on V. Similarly for the type Cn, 9 ~ S{V ® V) and A(V ® V) contain a trivial representation, which defines an antisymmetric form (,) on V. The Lie bracket can be described as a certain intertwining operator from 9 ® 9 to g.

Next we define Clifford and Weyl algebras. We denote wO = 1/2(El + ... + En -l + En), WI = 1/2{El + ... + En -l - En) and consider irreducible representations A ° and A l with highest weights WO and WI for the type Dn and irreducible representations SO and SI with highest weights -wo and _wI for the type Cn. The representations A ° and A l can be viewed as abstract half-spinor representations and SO and SI as abstract half­oscillator representations. The representations A ° ® V and A l ® V are decomposed into two nonequivalent components such that one of them is isomorphic to A 1 and A 0 , respectively (see also the tables in [VO]). Thus we can define two nonzero intertwining operators:

(1.2)

Fixing an element v E V yields two linear maps WO{v) : AO --t Al and w1(v) : Al --t AO. A similar decomposition also exists in the symplectic case. We denote by

(1.3)

the nonzero intertwining operators and, for v E V, we define the cor­responding linear maps <J>°(v) and <J>1(v). In order to find the relations between the linear maps, we note first that Ai ® S(V ® V) contains a com­ponent isomorphic to N, i = 0,1, with multiplicity one and similarly Si has multiplicity one in Si ® A(V ® V), i = 0,1 (see tables in [VO]). Let us denote by 6 the symmetrizer on V ® V, i.e. the projector to the subspace S(V ® V), and let 21 be the antisymmetrizer on V ® V. Then it follows that Wl(WO ® 1)(1 ® 6) is a natural projector of AO ® V ® V to AO given by the symmetric form 1 ® (,) and also WO{w l ® 1)(1 ® 6) is equal to 1 ® (,) as a map from Al ® V ® V to A I . This implies the anticommutation relations on A

(1.4)

130 J. Ding and 1. B. Frenkel

where '11 = '110 + '11 1 as an endomorphism of A = A ° E9 A 1• The factor algebra of the tensor algebra TV by the relation (1.4) is by definition the Clifford algebra associated to a nondegenerate bilinear symmetric form (,) on V. Similarly by considering the antisymmetrizer 2l on V ® V, we deduce that «1>1(<<1>0 ® 1)(1 ® 2l) = I ® (,) as a linear map from SO ® V ® V to SO and «1>0(<<1>1 ® 1)(1 ® 2l) = I ® (,) as a linear map from S1 ® V ® V to SI. This gives the following commutation relations on S

(1.5)

where «1> = «1>0 + «1>1 is an endomorphism of S = SO E9 S1. Now we can recover the spinor representation 'irA and oscillator repre­

sentation'lrs for orthogonal and symplectic Lie algebras in A and S:

where VI, V2 E V. It is certainly more standard to start from definitions of spinor and oscillator representations by formulas (1.6) and (1.7) and check directly the commutation relations for the classical Lie algebras using formulas (1.4) and (1.5).

Let V = V+ E9 V_ be a polarization of the fundamental representation corresponding to the weight decomposition into {fi}i=1 and {-fi}i=1 of the weights of V. One can consider the restriction of the spinor (1.6) and oscillator (1. 7) representations to the subalgebra gl{ n) ~ V+ ® V_ by choos­ing VI E V+, V2 E V_. Then there is a natural identification of the spinor representation A with the exterior algebra A * (V+) and the oscillator repre­sentation S with the symmetric algebra S*(V+) as representations of glen). In fact, the standard multiplication in A * (V+) and S* (V+) also has a similar interpretation in terms of the representation theory of glen). Thus we ar­rive at the well known classical construction of even Clifford/Weyl algebras and spinor/oscillator representations from the fundamental representation of glen). The odd Clifford and "odd Weyl" algebras together with spinor and oscillator representation of Lie algebras of type Bn and Lie superalge­bras of type Ben can be derived in the same way using 2n + 1 dimensional fundamental representation. The polarization V = V+ E9 VO E9 V- in the odd case contains an extra one-dimensional representation Yo, which adds an extra generator to the even Clifford or Weyl algebras.

In our review of the classical constructions, we only used the structure of the category of highest weight representations of g and the realization

Spinor and Oscillator Representations of Quantum Groups 131

of 9 as a symmetric or antisymmetric square of the fundamental represen­tation V. In the case of quantum group Uq(g), we can use Lusztig's result on q-deformation of category of representations and Faddeev-Reshetikhin­Takhtajan (FRT) realization of Uq(g) via a canonical solution of the Yang Baxter equation in V together with the quantum symmetric and antisym­metric form on V. Thus using the category of representations of quantum group, we are able to reproduce all the above steps in the q-deformed case. The quantum analogues of (1.6) and (1.7) yield the linear space of opera­tors corresponding to the quantum counterparts of A(V ® V) and S(V ® V). FRT realization uses quantum analogues of Borel subalgebras b± E 9 rather than 9 itself. Since, in the quantum case, additive structures are replaced by multiplicative ones, to obtain spinor and oscillator representations of quantum groups in FRT realization, one needs to perform Gauss decompo­sition of the right hand sides of (1.6) and (1.7). Finally to produce explicit formula for the standard generators K;l, Ei, Fi , i = 1, ... ,rank(g) of the quantum group, one has to take diagonal and adjacent to the diagonal el­ements in the Gauss decomposition. We would like to emphasize that all the structural information that we need for construction of quantum Clif­fordjWeyl algebras and spinor j oscillator representations as in the classical case is encoded into the fundamental n-dimensional representation of the quantum group Uq(g[(n». It consists of a choice of ordered basis {ei}i=l in Cn and the famous solution (5.1.1) of the Yang-Baxter equation in End(Cn ®Cn ), which also yields a representation of Hecke algebras Hm(q), m = 1,2,3, .... The representation-theoretic approach outlined above in the classical case brings a conceptional meaning to the numerous formulas of this beautiful theory.

We recall the basic facts about quantum groups in Section 2. Then in Section 3, we define quantum Clifford and Weyl algebras and construct spinor and oscillator representations for the types Bn, en and Dn. In Sec­tion 4, we consider the quantum group Uq(g[(n» and alternative definitions of the algebras and representations studied in Section 3. In particular, we obtain realizations of spinor and oscillator representations as exterior and symmetric algebras. In Section 5, we choose a canonical basis in the fun­damental representation and express R-matrix and quantum Clifford and Weyl algebras in this basis. This yields an explicit q-boson and q-fermion realizations, which were studied in the physics literature (See e.g. [PWI, [WZI, [KI, [IPI ). We also obtain explicit isomorphisms between quantum Clifford and Weyl algebras and their classical counterparts. This immedi­ately leads to realizations of quantum groups by endomorphisms of classical exterior and symmetric algebras. In Section 6, we will discuss possible fu­ture research directions.

132 J. Ding and I. B. Frenkel

2. Quantum groups and their representations

2.1 Quantum groups. The definition of quantum groups associated to Kac-Moody algebras was given by Drinfield [D1] and Jimbo [J1]. We will consider only the subclass of finite-dimensional simple Lie algebras.

Definition 2.1.1 Let (aij) be a Garian matrix of finite type of rank nand let di E {1,2,3} such that (diaij) is symmetric. Let q be an indetermi­nate. The quantum group Uq(g) is an associative algebra over C(q) with generators Ei , Fi , K i , K i- 1, (1 ~ i ~ n) satisfying the following relations:

KiKj = KjKi KiKi- 1 = K i- 1 Ki = 1

KiEj = qd,aij EjKi , KiFj = q-diaij FjKi

K- - K-:-l EiEj - FjEi = Oij d~ 3 d. q • - q- •

L (-1)S [l-saij ] E[EjEi =0, r+s=l-aij d.

if i =1= j,

L (_l)s[l-saij ] F[FjFt=O, r+s=l-aij ~

if i =1= j, (2.1.1)

where for integers N, M, d ~ 0, we define

N da -da IT q -q [N]d! = d -d'

a=l q - q (2.1.2)

The quantum group Uq(g) has a noncommutative Hopf algebra structure with comultiplication ~, antipode Sand counit c: defined by

~(Ei) = Ei ® 1 + Ki ® Ei ,

~(Fi) = Fi ® K;l + 1 ® Fi ,

~(Ki) = Ki ® K i,

S(Ei) = _K;l Ei,

c:(Ei) = c:(Fd = 0,

S(Fi) = -FiKi'

c:(Ki) = 1. (2.13)

We will use Uq(n+) to denote the subalgebra generated by Ei'S and Uq(n-) the subalgebra generated by Fi'S. We will also use U~(n±) to denote the subalgebra of Uq(n±) without unit, respectively.

Spinor and Oscillator Representations of Quantum Groups 133

Let Q = E&i=l Zai be a root lattice and let <, > be a bilinear symmetric form on Q such that < ai,aj >= diaij. We define K~ = Ki1 ••• K::'" for J.L = mlal + ... + mnan E Q, mi E Z. Then

K F · - q-<~.Otj>F·K ~ 3 - 3 W (2.1.4)

One can consider a more general quantum group than Uq(g) by replac­ing the lattice Q by a bigger lattice X such that Q c X c P, where P is the weight lattice. Moreover one can consider quantum group corresponding to a reductive Lie algebra 9 by choosing a lattice X C ~*, whose projection to ~: the dual of Cartan subalgebra of semisimple part of 9 satisfies the above condition. In particular, we define Uq(g!(n» by choosing X = E&~lZEi'

< Ei,Ej >= Dij so that Q = {Ei=lmiEiIE~lmi = O} and the projection of X to ~: coincides with P ~ Q'. Clearly the element K£, E = Ei=lEi, is central in Uq(g!(n». This definition was first given in [J2].

It follows from the definition of Uq(g) that S2(a) = q2Paq-2p , where a is an element in Uq(g) and p is one half of the sum of positive roots.

Definition 2.1.2 Let V be a module of Uq(g) and V* be its dual space. The dual module of V is defined as the space V* with the following action of Uq(g):

< llV, (a)v*, v >=< v*,1Tv(S(a»v >, (2.1.5)

where v* E V*, v E V, S is the antipode ofUq(g) and a E Uq(g).

Proposition 2.1.2 Let V be a module of Uq(g). Then V** is identified with V as a vector space and the action of a E Uq(g) on v·· is given by the action of q2Paq-2p on V.

The appropriate replacement of the cocommutativity of the classical universal enveloping algebra U(g) is the quasi-triangular structure of the quantum group Uq(g) discovered by Drinfeld [D2]. In order to introduce the quasi-triangular structure, we need to extend the ground field of the definition CCq) to formal power series C([q - Ill. We will also consider a completion Uq(g) in (q - I)-adic topology. Then, in particular, one can define t = log(q) = log(I+(q-I» and Hi = log(Ki)/t = log(I+ (Ki -I»/t. Note that Drinfeld [D2] used an equivalent approach by defining Ut(g) with generators Hi, Ei, Fi , i = 1, ... ,n, over C[[tll in the t-adic completion. Drinfeld obtained the following result.

Proposition 2.1.3 There is an element Vl inside Uq(g)®Uq(g) called the universal R-matrix for Uq(g), such that

(2.1.6)

134 J. Ding and I. B. Frenkel

with fi in U~(n+) and gi in U~(n-). It satisfies the following relations

Vt~(a) = ~OP(a)Vt,

(~ ® id)(Vt) = Vt13Vt23,

(id ® ~)(Vt) = Vt13Vt12 ,

where a E Uq(g), ~op denotes the opposite comultiplication,

Vt12 = L ai ® bi ® 1 = Vt ® 1, i

Vt13 = Lai ® 1 ®bi ,

i

Vt23 = L 1 ® ai ® bi = 1 ® Vt. i

Corollary 2.1.1. The following identities are valid

(8 ® 1)(Vt) = (Vt)-l,

(1 ® 8-1)(Vt) = (Vt)-l,

Vt12Vt13Vt23 = Vt23Vt13Vt12,

~(a)Vt21Vt12 = Vt21Vt12~(a),

where a E Uq(g) and Vt21 = Ebi ® ai.

(2.1. 7)

(2.1.8)

The third identity is known as the Yang-Baxter equation, the last identity is an intertwining property of the operator Vt21Vt12 and will play an important role throughout the paper. It is natural to call this element the universal Casimir operator.

Definition 2.1.3. Let 7l'v be a representation of Uq(g) in a finite dimen­sional space V. We set

L+V = (id®7rv)(Vt21),

L -v = (id ® 7rv ) (Vtll ),

LV = (id ® 7rV)(Vt21Vt12 ),

which are elements in Uq(g) ® End(V).

(2.1.9)

For any two representations 7rv, 7rw of Uq(g), we denote RVW

(7rV®7rw)(Vt).

Spinor and Oscillator Representations of Quantum Groups 135

Let V be a finite dimensional module of Uq(g). We will denote the im­age of U~(n±) in End(V) by End'±(V), respectively. Then the intersection

of End'+(V) and End'-(V) is zero. We denote by End°(V) the diagonal endomorphisms in the weight decomposition of V and let End±(V) be a direct sum of EndO(V) and End'±(V), respectively. For any A EEnd±(V), we denote by diag A its component in EndO(V). It is instructive to note that one can choose an appropriate basis in V consistent with the weight decomposition so that the elements of End±(V) are represented respec­tively by upper and lower triangular matrices; however, we do not need to use this fact.

Proposition 2.1.4. For any finite dimensional module V of Uq(g), L± E Uq(g) 0 End±(V) and the following identities are valid in Uq(g) 0 End(V) 0 End(V)

RvvLt.vL~V = L~VLrVRVV, RVV LtV L2V = L2V LtV RVv ,

LY R~t LrcRrt)-l = (RVV)-l Lr RVV Lr,

diag(L+V)diag(L-v) = I.

(2.1.10)

Proof. The first statement follows from (2.1.6). The Yang-Baxter identity in (2.1.8) implies

which proves the first two identities. By Definition 2.1.3, we have

(2.1.11)

From the identities above, we obtain that

136 J. Ding and I. B. Frenkel

Then

L[ R%tV Lf (R~V)-I = LtV (Llv)-I Rft LtV (L;V)-I(R~V)-I

= LtV LtV Rft (L1V)-I(L;v)-I(R~v)-1

= LtV LtV (L;V)-I(L1V)-1

= (RVV)-I LtV LtV RVV (L;V)-I(L1V)-1

= (RVV)-I LtV (L;V)-I RVV LtV (Llv)-I

= (RVV)-I Lf RVV L[.

The last identity follows from the explicit form of the universal R-matrix given in (2.1.6).

From Corollary 2.1.1, we obtain, for a E Uq(g),

«id ® 7rv )~(a))Lv = LV «id ® 7rv )~(a)). (2.1.12)

For any finite dimensional module V, the trace of Lq2p is a central element. This follows from Proposition 2.1.4.

When V is the fundamental representation of the quantum groups of types An, Bn, Cn. Dn, we will denote the operator matrices L±v and LV by L± and L, respectively.

Let V be a finite dimensional module of Uq(g) and let V 9:! V*. Then V is called a selfdual module. A selfdual module V has a nondegenerate invariant bilinear form (,). Let us fix an isomorphism J : V -+ V* and define an element C EEnd(V) by (CVI,V2) =< JVI,V2 >, for VI, V2 E V. For any element A in End(V), we denote At its transposition with respect to the form (,), i.e. (AtvI, V2) = (VI, AV2).

Proposition 2.1.5. Let V be a sel/dual module 0/ Uq(g), let us fix an isomorphism V 9:! V* denoted by J and define C E End(V) as above. Then

7rv(S-1 (a)) = C(7rv(a))tC- I ,

L±v C(L±V)tC-I = 1. (2.1.13)

2.2. Realization of quantum groups in fundamental representa­tions. Faddeev, Reshetikhin, Takhtajan [FRT] gave a quantum analogue of the matrix realization for the quantum groups of the classical types us­ing upper and lower triangular matrices. As in the undeformed case, the construction is based on the properties of the smallest fundamental repre­sentation V of the quantum groups of types An, B n, Cn, Dn. One needs a

Spinor and Oscillator Representations of Quantum Groups 137

triangular decomposition

End V = End' + (V) EI1 Endo (V) EI1 End' - (V)

and for the types Bn, an, Dn an element a E End(V) defined in the previ­ous subsection. Note that for the latter types, V is selfdual.

Definition 2.2.1. Let {b~} be a basis of End±(V), respectively. Then U (.c±) is an associative algebm with genemtors {l~} and the relations ex­pressed in terms of genemting functions .c± = Erl~ ® b~

{ Rvv.ct £~ =.c~ £t RVv,

Rvv.ct £2 = £2 £t RVv,

for Uq(o(N»,N E Z,N ~ 3, and Uq(sp(N»,N E 2Z,N ~ 2,

where a is an element in End (V) as above.

(2.2.1)

(2.2.2)

(2.2.3)

The algebra U(.c±) is a Hopf algebra with comultiplication, counit and antipode given by the following formulas

~(.c±) = .c±®.c±,

€(.c±) = I,

S(.c±) = (.c±)-l,

(2.2.4)

where ® assumes the multiplication in End(V). The invertibility of .c± follows from the invertibility of the diagonal elements (2.2.3).

Now comparing Definition 2.2.1 with properties of L±, we obtain a homomorphism

Moreover we have

Theorem 2.2.1. T defines an isomorphism of Hopf algebms U(.c±) and Uq(g) for the cases of glen), sp(2n) and o(N).

138 J. Ding and I. B. Frenkel

Proof. The surjectivity follows from the explicit formulas for the gen­erators of Uq(g) presented in [FRT]. The proof of injectivity for the case of gr(n) is given in [DF]. The proof for the case of sp(2n) and o(N) is essentially the same.

Besides .c±, we introduce full generating functions denoted by .c.

(2.2.5)

The element .c also has an inverse .c-1 = .c-(.c+)-1.

2.3. Representations of quantum groups. Let V be a representation of a quantum group Uq(g). For any Ji- E P, we define a weight subspace V(Ji-) = {v E VIKiv = q<!',l>;?v} and consider the category of diagonaliz­able highest weight representations (of type I) by analogy with the classical case. From a general result of Lusztig [L], it follows that this category is a q-deformation of its classical counterpart. More specifically, we have:

Proposition 2.3.1. [L] For any irreducible highest weight representation

VA' A E P, of g, there exists an irreducible highest weight representation Vi of Uq(g) (of type J) with the same character as that of VA of g.

Moreover if a tensor product VA 18) VI" A, Ji- E P, is completely reducible, the same is true for the quantum analogue. In this paper we will need only this case and we will assume the complete reducibility in this section. The role of the Casimir operator for U(g) is played by the universal Casimir operator 9t219t12 for Uq(g). In particular, Drinfeld has shown:

Proposition 2.3.2. [D3] Let ViI and Vi2 be irreducible highest weight rep­resentations of Uq(g). Then (1I"v q 18)1I"vq )(9t219t12) acts on the irreducible "1 "2 components with highest weight A3 (if they exist) of ViI 18) Vi2 as multipli-cation by q-( <AbA 1 +2P>+<A2,A2+2P>-<A3,A3+2p», where 2p is the sum of

positive roots and the form on ~* is fixed by the condition that the norm

square of the minimal root is 2.

If A1 = A2 = A, we would like to define an analogue of the permutation operator

(2.3.1)

where P is the standard permutation. One of the major differences between the classical and quantum case is that (RU ? = (1I"v q 18)1I"vq )(9l219t12) in "1 "2 general is not an identity operator. Proposition 2.3.2 implies

Corollary 2.3.1. The eigenvalues of RAA can only be

±q-1/2(2<A,A+2p>-<!J,!'+2p» ,

Spinor and Oscillator Representations of Quantum Groups 139

where J.L is the highest weight of an irreducible representation V/1 that occurs

in Vi®Vi.

Definition 2.3.1. The quantum symmetric subspace Sq {Vi ®Vi) is defined as the subspace of Vi ® Vi spanned by the eigenvectors of k)')' with eigen­values q-i(2<)',)'+2p>-<!,,!,+2p» for any possible J.L. The quantum antisym­

metric subspace Aq (Vi ® Vi) is defined as the subspace of Vi ® Vi spanned by the eigenvectors of k)')' with eigenvalues _q-i(2<)".>.+p>-<!',!'+p» for

any possible J.L.

Then we have

(2.3.2)

Let V and W be two diagonalizable highest weight modules of Uq{g). We define a diagonal operator D WV on Wand Vasa multiplication by q<)',!,> on W(A) ® V{J.L). Then we have from (2.1.6)

R WV = Dwv {l+ Exi®Yi), Yi EEnd'-(V) (2.3.3)

and similarly

(2.3.4)

Certainly we also have Xi E End'+{V) and Xi E End'-{V), though it is not relevant for the following

Proposition 2.3.3. Let k~v and Rr{v be any elements of End(W) ® End{V) satisfying conditions (2.3.3) and (2.3.4), repectively, such that

;;,WVR-WV _ RWVRWV . ""21 12 - 21 12'

then Rr{v = ~"{v and R~v = R~v.

(2.3.5)

Proof. The proof immediately follows from the fact that the intersection of End'+{V) and End'-{V) is zero.

2.4. Properties of R-matrices in fundamental representations. Now we will identify quantum symmetric and antisymmetric spaces for the fundamental representation of quantum groups corresponding to the infinite series An, Bn, en, Dn.

We will start with Uq{s!(n». We denote the n-dimensional irreducible fundamental representation with the highest weight Wi by V~ =Cn . As in the classical case, V~ ® V~ is decomposed into two irreducible components,

140 J. Ding and I. B. Frenkel

namely Aq(V~ ® v~) and Sq(V~ ® v~). Computation of the eigenvalues of RWIWI yields respectively, _q-l-l/n and ql-l/n. If we consider the quan­tum group Uq(gl(n)), the contribution of KE to R-matrix will be equal to ql/n. We denote by R the operator RWIWI for the quantum group Uq(gl(n)) rather than for Uq($l(n)). We obtain

Proposition 2.4.1. R acts on Aq(V~ ® v~) as multiplication by the con­stant _q-l and on Sq(V~ ® V~) by q. Moreover we have

(R - qI)(R + q-1I) = O. (2.4.1)

The proof of (2.4.1) follows from the fact that R - qI and R + q-l I annihilate Sq(V~ ® v~) and Aq(V~ ® v~), respectively.

Next we consider the quantum group Uq(o(N)) and its N-dimensional fundamental representation with the highest weight WI denoted by vq. The classical theory asserts that V q ® V q has three irreducible components, Aq(Vq ® V q), one dimensional trivial representation and its compliment S~(vq ® V q) in the quantum symmetric space.

A direct computation of eigenvalues of R = RWIWI implies

Proposition 2.4.2. In the case of Uq(o(N», R acts on Aq(V~ ® V~) as multiplication by the constant _q-l, on S~(vq ® V q) by q and on the one dimensional trivial representation by ql-N.

Moreover we have

(R - qI)(R + q-1I)(R - ql-N J) = O. (2.4.2)

The proof of (2.4.1) is similar to that of Proposition 2.4.1.

Corollary 2.4.1. Let K =1 - (1/(q - q-I»(R-I - R). Then K projects the space V q ® V q to the one-dimensional space and one has

(2.4.3)

The image of K is the one-dimensional invariant subspace in V q ® V q

corresponding to the trivial subrepresentation. As in the classical theory, the symplectic quantum group Uq(sp(N)),

N = 2n, can be treated in a parallel to the case of the even dimensional orthogonal quantum group. We denote again by V q the N-dimensional fundamental representation with the highest weight WI. Then vq ® V q

has three irreducible components, Sq(Vq ® V q), the one-dimensional trivial

Spinor and Oscillator Representations of Quantum Groups 141

representation and its complement in the quantum antisymmetric subspace A~«Vq ® vq). We denote R = RWtWt and we obtain the following

Proposition 2.4.3. For Uq(sl'(2N», N = 2n, R acts on Aq(V~ ® V~) as multiplication by the constant _q-l, on S~(vq ® V q) by q and on the one-dimensional trivial representation by _q-l-N and

(R - qI)(R + q-1I)(R + q-l-N I) = o. (2.4.4)

Corollary 2.4.2. Let K =1 - (1/(q - q-l»(R-1 - R). Then K projects the space V q ® V q to the one-dimensional space and

2 _ q - q-l + q-l-N _ qN+l K - 1 K. q-q-

(2.4.5)

The image of K is the one-dimensional invariant subspace of vq ® vq

corresponding to the trivial subrepresentation.

Proposition 2.4.4. Let L' = L - I. Then for the case of gl(n)

for the case of o(N) or sl'(2n)

L~R21L~R2l + (q - q-l)(p - K)L;R2l

= R-1 L~RL~ + (q - q-l)R-1 L~(P - K),

where K = PK and K = KP.

Proof. For Uq(gl(n», we have

or

L~R21L~R71 + R21L~R2l- R-1 L~R2l = R-1 L~RL~ + R-1 L~R - R-1 L~R2l.

Using the quadratic relation (2.4.1), we obtain

(2.4.7)

142 J. Ding and I. B. Frenkel

The proof of (2.4.7) follows from the definition of K.

3. Quantum Clifford algebras and Weyl algebras;

spinor and oscillator representations

3.1. Quantum Clifford algebra and spinor representation in even dimensions. In this subsection, we introduce the notion of the quantum Clifford algebra. This is done in an abstract way using the representation theory of the quantum group Uq(o(N)), N = 2n. When q = 1, one recovers the classical Clifford algebra as was explained in the introduction. Then using the quantum Clifford algebra, we will construct the spinor represen­tation of quantum group Uq(o(N)), in the Faddeev-Reshetikhin-Takhtajan realization. The odd dimensional case N = 2n + 1 requires a minor modi­fication of this construction and will be considered in the next subsection.

Let V q be a 2n-dimensional fundamental representation of Uq(o(2n)). We denote by (,) an invariant bilinear form on V q • We will also use the same notation for roots and weights as in the classical case described in the introduction. Lusztig's theorem about deformation of the category of highest weight representations implies the following:

Proposition 3.1.1. Let A~ be an irreducible module ofUq(o(2n)) with the highest weight wO and let A~ be an irreducible module of Uq(o(2n)) with the highest weight WI. The module A~ 0 V q and the module A~ 0 V q are decomposed into two nonidentical irreducible components. There exist two nonzero intertwining operators WO and WI:

WO : A~ 0 V q -4 A!,

WI : A! 0 V q -4 A~.

The operators WO and WI are unique up to a scalar multiple.

(3.1.1)

The module V q 0 vq has two invariant subspaces, the quantum symmetric space Sq(Vq 0 V q) and the quantum antisymmetric space Aq(Vq 0 vq). The module Sq(Vq 0 V q) can be decomposed into two irreducible components, one of which is the one-dimensional trivial rep­resentation.

Let 6 q be a quantum symmetrizer on V q 0 V q, which maps V q 0 V q

onto Sq(Vq 0 vq).

Proposition 3.1.2. Let WO, WI be defined as above for Uq(o(2n)). Then Wl(W00I)(I06q) is a nonzero intertwining operator from A~0Vq0Vq to

Spinor and Oscillator Representations of Quantum Groups 143

A! given by the invariant form I®(,), andwO(w1®I)(I®6q) is a nonzero intertwining operator from A~ ® v q ® v q to A~ also given by I ® (,).

Proof. The tensor product A! ® V q decomposes into two irreducible com­ponents. This implies that there are two copies of irreducible components in A! ® v q ® V q which are isomorphic to A!. The module A! ® V q ® V q

is isomorphic to A! ® Sq(Vq ® V q) E9 A! ® Aq(Vq ® vq). The module A! ® Aq(Vq ® V q) contains an irreducible component isomorphic to A!. Thus the map from A! ® Sq(Vq ® V q) to A! can only be the intertwining operator I ® (,). On the other hand, the operator '111('110 ® 1)(1 ® 6 q ) is an intertwining operator by construction; thus we prove the first assertion of the proposition.

The second assertion is proved by a similar argument. We note that in the above proposition, the normalization of the form is

chosen to be consistent with the normalization of the intertwining operators '111 and '110. We will assume from now on that this relation always holds. We define Aq = A~ E9 A! and'll = '110 + '111. Then'll is an intertwining operator from Aq®Vq to Aq. For any vector v in vq, we obtain an operator w(v) in End(Aq). Proposition 3.1.2 implies

Corollary 3.1.1. Let {Vi} be a basis ofVq, and R:/ be the matrix coeffi­cients of R = Rvqvq . Then

In (3.1.2). we assumed summation over repeated indexes i', j'. This motivates the following

(3.1.2)

Definition 3.1.1. Even dimensional quantum Clifford algebra Cq(2n) is the tensor algebra of v q ~ C2n factor by the relations

(3.1.3)

where VI and V2 are elements in V q.

Theorem 3.1.1. Even dimensional quantum Clifford algebra Cq(2n) is isomorphic to the algebra End(Aq).

Proof. The definition of the quantum Clifford algebra implies that there is a surjective homomorphism from this algebra to the algebra generated by w(v). From the concrete formula of the commutation relations of the generators in Section 5 in the specific basis, it follows that the ordered

144 J. Ding and I. B. Frenkel

monomonials of the generators span the whole algebra. Thus the dimen­sion of the quantum Clifford algebra is less or equal to 22n. This implies injectivity.

Now we will proceed to construct the spinor representation of quantum group Uq(o(2n)) using the quantum Clifford algebra. Our first goal is to find an expression for 7rM (,C), which is an intertwining operator from Aq ® V q to itself. We can easily construct two intertwining operators in the same space using the quantum Clifford algebra. Let F : C - V q ® V q

be an intertwining operator. Then both \II 0 F 0 \II and I are intertwining operators on Aq ® V q. We choose the normalization of F as follows. Let {Vi} be a basis in V q and let {vi} be a dual basis in V q*. We denote by F' = E( Vi, Vj }vi ® vi so that F' P is an invariant element in V q* ® v q* such that < F',F >= tr(q2p }, where < vi ®Vi,Vl ®V2 >=< vi,Vl >< Vi,V2 >, for Vi, V2 E vq, vi, vi E vq*. We can also view F as a linear map from C to V q ® vq • Then we have

Proposition 3.1.3. The spinor representation of the element L E Uq(o(N)) ® EndVq, N = 2n, is given by

7rAq (L} = a(b\ll 0 F 0 \II + I),

h q b - _( _ -1) 1-N W ere a = 1 (q_q-l)ql-N/(1+q2-N) , - q qq.

Proof. We recall first that the tensor product A~ ® vq, i = 0, I, has exactly two irreducible components. This immediately implies that the intertwining operator 7rAi (L) is a linear combination of \IIioFo\lli', {i, i/} =

q

{O, I}, and 1. The constants can be found by comparing the action of the both sides on the highest weight vectors. The calculation will be explained by the explicit formulas later.

Theorem 3.1.2. There exists a unique decomposition of 7rAq (L} of the form A+(A-}-l, where A+ = D(l + EXi ® Yi}, A- = D(l + Ex~ ® yD, Yi E Encl'-(Vq), y~ E End'+(Vq) and D is a diagonal operator in Aq ® V q

with entries q±1/2. The spinor representation of Uq('c±) is given as

(3.1.4)

Proof. The statement follows from Proposition 2.3.3 and the expression (2.1.6) for the universal R-matrix.

Constructing the spinor representation of the quantum group Uq (0 (2n» was derived in a purely abstract way. However it is not difficult to obtain a

Spinor and Oscillator Representations of Quantum Groups 145

direct verification of Proposition 3.1.3. In the classical case, the verification of the spinor reprsentation immediately follows from the anticommutation relations for generators of Clifford algebra. The direct proof is also impor­tant since it can be extended to more general quantum Clifford algebras defined in terms of any representation of Birman-Wenzl algebra [BW].

In order to perform this derivation, we will introduce a certain basis free notation widely used in the physics literature. First, we define a dual intertwining operator

It will be convenient to identify W with an element EW(Vi) ® vi in End(Aq) ® V q*, where {Vi} is a basis in V q and {Vi} is a dual basis in V q*. We also identify w* with as an element in End(Aq) ® V q • We will use this notation in the rest of this subsec­tion. For any element A in End(Vq), we have a natural left action of A on V q and a natural right action on V q* defined by the formula < v* A, v >=< v*, Av >, v E V q , v* E V q*. Finally we will use the nota­tion similar to the one in Proposition 2.1.4. Namely we denote by WI W2 an element in End(Aq) ® V q* ® V q* defined by EW(Vi)W(vi) ®vi ®vi , i.e. we assume multiplication in End(Aq). Similarly W2Wl denotes an element in End(Aq) ® V q* ® V q* defined by EW(Vj)W(Vi) ® vi ® vi. We also assume that R12 acts on V q ® V q from the left and on V q* ® vq* from the right. Now we can rewrite the commutation relation in a basis free form.

Proposition 3.1.4. The commutation relations of w* and W are

W2Wl = WI W2( _q-l R211) + gF',

WtW2 = W2( _q-l R 12 )W! + g1,

,T.*,T.* -IR-1,T.*,T.* + F ':1.'1':1.'2 = -q 12 ':1.'2':1.'1 g,

(3.1.5)

where F and F' are defined above and 9 = qN-2, N = 2n. 1 denotes the identity element in V q ® V q* corresponding to identity in End(Vq).

Proof. The first identity is a basis-free form of (3.1.2). The second and the third identities follow from the first identity and the definition of w* .

Proposition 3.1.5. Let 1/ = W*W. Then

146 J. Ding and I. B. Frenkel

Proof.

wi'll 1 (_q-l R21 )W;W2( _q-l R'211)

= WiW;WIW2(-q-l R2/) - gWiIW2(-q-l R2/) = _q-l R12 W;WiW 1 W2( _q-l R'2l) + gFW I W2( _q-l R'2l)

- gWiIW2( _q-l Rii)

= g(_q-l Rdw;wiF' + (_q-l RI2)W;Wi W2Wl + gFWIW2(-q-l Rii)

- gwi IW2(-q-1Ri21)

= g( _q-l R12)W;Wi F' + (_q-l R12)W;W2( _q-l R12)wi wl

- g(_q-l R 12 )W;IW I + gFW IW2(-q-1 Rii) - gwiIW2(-q-l Rii)

= _gq-l R12W;IwlK + (_q-l R12)W;W2( _q-l RdWiWl

- g(_q-l R I2 )W;W2P + gKwi IW2(-q-1 Rii) - gPW;W2(-q-1 Rii)

= _gq-l R I2 W;W2 PK + (_q-l RI2)W;W2( _q-l RI2)WiWl

- g( _q-l Rdw;W2P + gK Pili; 1li2 ( _q-l Rii) - gPIli;W2( _q-l Rii).

The equalities WiF' = IwlK and FiliI = KlliiI follow from the re­lation K =p(F ® F'), where p permutes the middle two components in V q ® V q ® V q* ® V q*. This relation follows from the definitions of F, F' and K. It can also be verified in the concrete basis in Section 5.

Corollary 3.1.3. Let t = a(bw*w+I), where a and b are as in Proposition 3.1.3. Then

(3.1.7)

Proof. The identity follows immmediately from Proposition 3.1.4 and Proposition 2.4.4. The constant b is fixed by relation (3.1.7) alone. The constant a can be calculated from the second formula of (3.1.5). In fact, the explicit action of '11*'11 on the tensor product of two highest weight vectors in Aq ® vq is equal to the constant g/(l + g). On the other hand, the action of ,C on the same vector in Aq ® V q is equal to q by Proposition 2.1.3.

Thus we are able to directly check that t satisfies the same relation as 7rAq (,C). It is substantially more difficult if not impossible to prove directly that the components of t in the Gauss decomposition provide a representation of .c±. We can only conclude this fact on the basis of the representation theory of quantum group Uq (o(2n)), Thus the spinor representation leads to a natural question of defining quantum group by means of operator ,C rather than operators .c±, as in [FRT]. The algebra generated by ,C can be viewed as a quantum analogue of Lie algebra. We

Spinor and Oscillator Representations of Quantum Groups 147

will study quantum Lie algebras and their representations in a subsequent paper.

3.2 Quantum Clifford algebra and spinor representation in odd dimensions. We will now consider the case of o(2n + 1), which is parallel with and in some sense easier than the even dimensional case, due to the irreducibilty of the spinor representation.

Proposition 3.2.1. Let Aq be an irreducible highest weight module with the highest weightwO, i.e., the representation ofo(2n+l). The module Aq®Vq

is decomposed into two nonidentical irreducible components. There exists a nonzero intertwining operator \lI :

(3.2.1)

This operator \lI is unique up to a scalar multiple.

As in the even dimensional case, the module V q ® V q decomposes into two invariant subspaces, the quantum symmetric space Sq(Vq®Vq) and the quantum antisymmetric space Aq(Vq ® V q), and the module Sq(Vq ® V q) has two irreducible components, one of which is the one-dimensional trivial representation. Let (,) be an invariant bilinear form on V q defined by the projection to the trivial representation. We also denote 6 q as the quantum symmetrizer on V q ® V q •

Proposition 3.2.2. Let \lI be defined as above for Uq (o(2n + 1)). Then \lI(\lI ® 1)(1 ® 6 q) is a nonzero intertwining operator from Aq ® vq ® vq

to Aq given by I ® (,).

Proof. The tensor product Aq ® Sq(Vq ® V q) contains Aq with multiplicity one. This implies that the two intertwining operators \lI{\lI ® 1)(1 ® 6 q )

and I ® (,) are proportional. The normalization of the form (,) is chosen to be consistent with the normalization of \lI.

Corollary 3.2.1. Let {Vi} be a basis of V q , and R~/ be the matrix coef­ficients of R = R vqvq . Then

(3.2.2)

Definition 3.2.1. Odd dimensional quantum Clifford algebra Cq (2n + 1) is the tensor algebra of vq ~ c2n+1 factor by the relations

(3.2.3)

148 J. Ding and I. B. Frenkel

where Vl and V2 are elements in vq.

Theorem 3.2.1. Odd dimensional quantum Clifford algebra Cq(2n + 1) is isomorphic to a direct sum of two copies of the algebra End(Aq).

Proof. The realization of the quantum Clifford algebra Cq (2n + 1) in the specific basis in Section 5 implies that it is isomorphic to the tensor product of Cq (2n) and the two-dimensional subalgebra generated by w(vo), where CVo E V q is the one-dimensional subrepresentation with respect to the quantum subgroup Uq(gl(n» of Uq(o(2n+ 1». Then the statement follows from Theorem 3.1.1.

Spinor representation of Uq (o(2n + 1» is constructed from the odd di­mensional quantum Clifford algebra in exactly the same way as in the even dimensional case considered in Section 3.1. In particular, the statements of Theorem 3.1.2 and Propositions 3.1.3-3.1.5 are valid with N = 2n + 1.

3.3. Quantum Weyl algebra and oscillator representations. In this subsection, we will introduce the notion of the quantum Weyl alge­bra and then construct the oscillator representation of the quantum group Uq(sp(2n». All the definitions and statements are completely parallel to the ones of Section 3.1. There is one important difference between spinor and oscillator representation, namely the former is finite dimensional while the latter is not. However in the quantum case, the weight decomposition of both representations is the same as in the classical case and by using the graded structure of representations, one can avoid possible complications of the infinite dimensional case.

Let V q be a 2n-dimensional fundamental representation of Uq(sp(2n». We denote by (,) an invariant form on vq as before.

Proposition 3.3.1. Let S~ be the irreducible module of the Uq(sp(2n» with highest weight -wo and S~ be the irreducible module of Uq(sp(2n» with highest weight _wl . The module S~ @ vq and the module S~ @ vq are decomposed into two nonidentical irreducible components. There exist two nonzero intertwining operators <1>0 and <1>1:

<1>0 : S~ @ V q ---+ S~,

<1>1 : S~ @ V q ---+ S~.

The operators <1>0 and <1>1 are unique up to a scalar multiple.

(3.3.1)

The module vq @ V q is decomposed into two invariant subspaces, the quantum symmetric space Sq(Vq @ V q) and the quantum antisymmetric

Spinor and Oscillator Representations of Quantum Groups 149

space Aq(Vq ® vq}. The module Aq(Vq ® V q} has two irreducible compo­nents, one of which is the one dimensional trivial representation.

Let l2tq be the quantum antisymmetrizer, which maps V q ® V q onto Aq(Vq ® vq).

Proposition 3.3.2. Let c}>o, c}>1 be defined as above for Uq(sp(2n}}. Then c}>O(c}>1 ® 1}(1 ® l2tq} is a nonzero intertwining operator from S~ ® V q ® V q

to S~, and c}>1(c}>O ® 1}(1 ® l2tq} is a nonzero intertwining operator from sg ® v q ® v q to sg. Both of them are given by 1 ® (,).

The proof is the same as that of Proposition 3.1.2. In the proposition above, as in the case of o(2n}, we actually normalize

the operators c}>1 and c}>0, so that they are consisitent with the form (,). We define Sq = sg ill S~ and c}> = c}>0 + c}>1. Then c}> is an intertwiner from Sq ® v q to Sq. For any v E V q, we denote by c}>(v} the corresponding element in End(Sq}.

Corollary 3.3.1. Let {Vi} be a basis ofVq, and R!,J' be the matrix coef­

ficients of R = Rvqvq . Then

(3.3.2)

Definition 3.3.1. Quantum Weyl algebra is defined as the tensor algebra of v q factor by the relations

(3.3.4)

where VI and V2 are elements in V q.

Theorem 3.3.1. The quantum Weyl algebra Wq(2n} is isomorphic to the subalgebra of End ( Sq} generated by c}> (v) for v E v q.

Proof. The injectivity can be derived from the explicit realization of the quantum Weyl algebra in the canonical basis using the same argument as in the proof of Theorem 3.1.1. Note that since the dimension of Wq (2n} is infinite, one should consider its graded dimension.

One can view the quantum Weyl algebra as an infinite matrix alge­bra. However, as in the classical case, the finite dimensional subspaces Hom(Sq(A}, Sq(J.L» c End(Sq}, where A, J.L are arbitary weights, under the above identification do not belong to the quantum Weyl algebra but only to its certain completion. To construct the oscillator representation of quan­tum groups Uq(sp(2n», we again find first a representation of the element

150 J. Ding and I. B. Frenkel

L and then consider its Gauss decomposition. Let F be an intertwining operator from C to V q ® V q defined as in Section 3.1. Then we have

Proposition 3.3.3. The oscillator representation of the element L E Uq(sp(2n» ® End(Vq) is given by

7rAq (L) = a(b\ll 0 F 0 \lI + I),

wherea= 1+q-l N(q :::)J(1+q-2-N) , b=q-l-N(q_q-l), N=2n.

Theorem 3.3.2. There exists a unique decomposition of 7rSq (L) of the form of A+(A-)-l, where A+ = DCl + EXi ® Yi), A- = D(l + Ex~ ® yD, Yi E End-(V) y~ E End+(V) and D is a diagonal matrix in Sq ® vq with entries q±r/2, r E Z. The representation of U('c±) is given as follows:

We can also develop a calculus for quantum Weyl algebra as we did for the quantum Clifford algebra in Section 3.1. We will again identify ~ with an element in End(Sq) ® vq* and ~* with an element in End(Sq) ® vq. Let us define elements F E V q ® V q, F' E V q* ® V q* and I as in Section 3.2. Then we obtain the following counterparts of Proposition 3.1.4-3.1.5 and Corollary 3.1.3.

Proposition 3.3.4. The commutation relations of ~ and ~* are

~i~2 = qRll~2~i + gF,

~i~2 = ~2(qR12)~i + gI,

~2~1 = ~1~2(qR2l) + gF',

where F, F' and I are defined above and g = q2n+2.

Proposition 3.3.5. Let L' = ~*~. Then

3.3.5

L~R21L~R2l + gq-l(p - k)L~R2l = R-1 L~RL~ + gq-l Rll L~(p - K). (3.3.6)

Spinor and Oscillator Representations of Quantum Groups

Proof.

<I>i <I> 1 (qR21 ) <I>; <I> 2 ( qR;:l)

= <I>i<I>;<I>1<I>2(qR2'"l) - g<I>i I <I>2(qR2l>

= qR12<I>; <I>i <I> 1 <I> 2 (qR2l> + gF<I>l <I> 2 (qR;l )

- g<I>iI<I>2(qR1l) = g(qR12 )<I>;<I>iF' + (qR12)<I>;<I>i<I>2<I>1 + gF<I>1<I>2(qR1l)

- g<I>i I <I> 2 (qRil)

= g(qR12 )<I>;<I>iF' + (qR12 )<I>i<I>2(qR12<I>i<I>1 - g(qR12 )<I>iI<I>1

+ gF<I>1<I>2(qR121) - g<I>iI<I>2(qR1l) = gqR12<I>;I<I>lK + (qR12 )<I>i<I>2(qR12<I>i<I>1 - g(qR12)<I>;<I>2P

+ gK<I>U<I>2(qR1l) - gP<I>;<I>2(qRp})

= gqR12<I>;<I>2PK + (qR12)<I>i<I>2(qRd<I>i<I>1 - g(qR12)<I>;<I>2P

+ gKP<I>i<I>2(qR1l) - gP<I>i<I>2(qR1l)·

151

Corollary 3.3.3. Let L = a(b<I>*<I> + 1), where a and b as in Proposition 3.3.3. Then

(3.3.7)

The calculation of a and b are derived in the same way as in Section 3.1.

Representation-theoretic arguments allows one to identify L with 71'Sq (.£). As we mentioned at the end of Section 3.1, the algebra gener­ated by .£ can be viewed as a quantum analogue of the Lie algebra sp(2n).

Finally, we note that there is a natural odd dimensional analogue of the quantum Weyl algebra that can be defined and studied in a parallel to the odd dimensional quantum Clifford algebra discussed in Section 3.2. This can be done using the representation theory of the quantum group of type BCn corresponding to a superalgebra when q = 1.

4. Polarization of quantum Clifford and Weyl algebras

and representation of Uq(g[(n»

4.1 Quantum group Uq(g[(n» as a sub algebra in Uq(o(2n» and Uq(sp(2n». Quantum group Uq(gl(n» can be imbebded into Uq(o(2n» or Uq(sp(2n» as a subalgebra, if we omit the generators En and Fn cor­responding to the last node of the Dynkin diagrams of types Dn or Cn.

152 J. Ding and I. B. Frenkel

We will denote this map by 19 , where 9 is Uq(o(2n» or Uq(.sp(2n». Thus we can restrict spinor and oscillator representations as well as other rep­resentations of Uq(o(2n» or Uq(.sp(2n» to its subalgebra Uq(gl(n». We will start with the fundamental representation vq of dimension 2n, The following statement is the q-deformation of the well-known classical facts.

Proposition 4.1.1. The fundamental representation vq of Uq(o(2n» or Uq(.sp(2n» is decomposed into two irreducible components Vf with respect to the subalgebra Uq(gl(n» of Uq(o(2n» or Uq (.sp(2n», where V': is the fundamental representation of Uq (gl( n» and V.! is the dual representation of V.:.

We can explicitly identify V.! with V.:* via the invariant form on vq. The form is degenerate on V': ® V': and V.! ® V.!

The decomposition vq = V': El7 V.! stated in the proposition above is called polarization. The polarization defines natural imbeddings End(Vf) -+End(Vq). We however will need another linear map:

given by i('lIv.;')(a) = 1Tv.(a), a E Uq(gl(n». The map i is well defined since the annihilators of V': and V.! coincide. It allows the imbedding of Uq(gl(n» into Uq(o(2n» or Uq(.sp(2n» in the invariant form of Faddeev­Reshetikhin-Takhtajan. We will define two maps:

where 9 = o(2n) or .sp(2n), by the following formula

(4.1.1)

where trq (x)=tr(xq2p ), x E End(V':).

Proposition 4.1.2. The maps J o(2n) and J sp(2n) are injective homomor­phisms of U(.c;I(n» and U(.c~2n» or U(.c;p(2n»' respectively. We also have

where 9 is Uq(o(2n» or Uq(.sp(2n», respectively.

Proof. As modules of Uq(gl(n», V': ® V':,V': ® V.! and V.! ® V.! have no common submodules between any two of them. Thus the restriction

Spinor and Oscillator Representations of Quantum Groups 153

of the operator PRV9V9 acting on vq ® vq to the subspace V~ ® V~ or V.! ® V.! will give us an intertwining operator on V~ ® V~ and V.! ® V.! for the subalgebra Uq(g(n)), respectively.

On the other hand, the restriction of RV9V9 to the subspace V~ ® V~ or V.! ® V.! also satisfies the Yang-Baxter equation. Due to the fact that subspace V~ ® V~ or V.! ® V.! as a module of Uq(g(n» has two irreducible components, we obtain that the restriction of RV9V9 to the subspace V~ ® V~ or V.! ® V.! must coincide with RV!{.v!{. or RV:!.v:!. of Uq(g(n)). Thus if we restrict the second factor of L±v9 to End(V~), we will obtain L±v!{., which is equivalent to the formula (4.1.1). The commutativity also follows from the above property of RV9V9.

4.2. Spinor and oscillator representations of Uq(g(n». Now we will consider the restriction of the spinor and oscillator representations to the quantum group Uq(g(n)). The commutation relation between the intertwining operators corresponding to the subspaces V~ and V.! can be written using the projection of the universal R-matrix for Uq(g(n)) onto the product of two n-dimensional representations en ® en.

Proposition 4.2.1. Let {Vi} be a basis of V~ and let {vi} be a basis of V.!. Let 'I/J(Vi) = \{I(Vi), 'I/J*(vi ) = \{I*(vi ), ¢(Vi) = <I>(Vi), ¢*(vi ) = <I>*(vi ).

Then

- qRtf 'I/J(vi') ® 'I/J(Vi') = 'I/J(vi)'I/J(vi)

- qR~;:i''I/J*(vi') ® 'I/J*(vi') = 'I/J* (vi)'I/J* (vi) (4.2.1)

*' -1 i' ,j * ., . . 'I/J(Vi)'I/J (v3) = -q Ri,i''I/J (v3 )'I/J(Vi') + (Vi, v3)J,

and

1 0,,/

q- R~,'/ ¢(vi') ® ¢(Vi') = ¢(vi)¢(vi)

q-l R!::i,¢*(vi') ® ¢*(vi') = ¢*(vi)¢*(vi) (4.2.2)

¢(vi)¢*(vi) = qR~>/¢*(vi')¢(vi') + (vi,vi)J,

where Rff are the matrix coefficients of the R-matrix for Uq(gl(n» on V~®V~.

The quantum Clifford algebra Cq(2n) is naturally isomorphic to an algebra with generators 'I/J(Vi) and 'I/J*(vi ) satisfying the relations (4.2.1). The quantum Weyl algebra Wq(2n) is naturally isomorphic to an algebra generated by ¢(Vi) and ¢*(vi) satisfying the relations (4.2.2).

The proof follows from the fact that the restriction of RV9 (jWQ to

154 J. Ding and I. B. Frenkel

V~ ® V~, V.! ® V.!, V~ ® V.! yields the correponding R-matrices of Uq(gl(n» and the degeneracy of the form (,) on the first and second tensor products.

Combining the linear maps 1/J(Vi) and 1/J(vi ) or ¢(Vi) and ¢(vi ), we obtain the intertwining operators: 1/J: Aq ® V~ -+ Aq, 1/J* : Aq ® V.! -+ Aq, ¢ : Sq ® V~ -+ Sq, ¢* : Sq ® V.! -+ Sq for the quantum group Uq(gl(n».

Definition 4.2.4. The quantum exterior algebra A;(V~) is a factor algebra of tensor algebra ofV~ modulo the relation (I +qPR21 )(V1 ®V2) = O. The quantum symmetric algebra S;(V~) is a factor algebra of tensor algebra of V~ modulo the relation (I - q-1PR21 )(V1 ® V2) = o.

Proposition 4.2.2. The quantum exterior algebra A;(V~) is isomorphic to Aq as a module over the Clifford algebra Cq(2n). Quantum symmetric algebra S;(V~) is isomorphic to Sq as a module over the Weyl algebra W q (2n).

We can also obtain spinor and oscillator representation of Uq(gl(n».

Proposition 4.2.3. Let L = LV::' be operator {2.1.9} for Uq(gl(n». Then

(4.2.3)

defines a representation of Uq(gl(n» on Aq, where a and b as in Section 3.1.

7rSq (L) = a(b¢* ¢ + I) (4.2.4)

defines a representation of Uq(gl(n» on Sq, where a and b as in Section 3.3.

Theorem 4.2.1. Let L = LV::' be as in Proposition 4.2.3. There exists a unique decomposition of7rAq(L) (resp. 7rsq(L» of the form A+(A-)-1,

where A+ = D(l + EXi ® Yi), A- = D(l + Ex~ ® yD, Yi E End'-(V~), y~ E End'+(V~) and D is a diagonal operator. The spinor {resp. oscillator} representation is given by 7rAq (L±) = A± (resp. 7rsq(L±) = A±).

By considering 1/J as an element of End(Aq) ® V.! and 1/J* as an element in EndAq ® V~, and similarly for ¢ and ¢*, we can rewrite the relations (4.2.1) and (4.2.2) in the basis free form.

Proposition 4.2.3. The relations between 1/J* and 1/J are

.1.*.1.* -1 (R )-1.1.*.1.* 'f'1 'f'2 = -q 12 'f'2 'f'1 ,

1/Jr1/J2 = 1/J2( _q-1 R 12 )1/Jr + gI,

1/J21/J1 = 1/J11/J2 (_q-1 (R2t}-1).

(4.2.5)

Spinor and Oscillator Representations of Quantum Groups 155

The relations between if>* and if> are

if>iif>; = q(R12 )-1if>;if>i,

if>iif>2 = if>2(qR12)if>i + gI if>2if>1 = if> 1 if>2(q(R21 )-1).

(4.2.6)

We can also compute the commutation relations among 'I/J*'I/J, 'I/J*'I/J* and 'I/J'I/J, and among if>*if>, if>*if>* and if>if>. In particular, we have

Proposition 4.2.4. Let £ = 'I/J*'I/J. Then

Let £ = if>*if>. Then

£lR21 £2(R2t}-1 + gq-l P£2(R21 )-1 = (R)-l £2R£1 + gq-l(R12 )-1 £2P. (4.2.8)

Proof.

'l/Ji'I/Jl (_q-l R2t}'I/J;'l/J2( _q-l (R21 )-1) = 'l/Ji'I/J;'l/Jl'I/J2g( _q-l (R21 )-l) -'l/Ji I'l/J2( -q-l(R21 )-1)

= _q-l (R12 )'I/J;'l/Ji'I/Jl'I/J2( _q-l (R21 )-l) - g'I/JU'l/J2( _q-l (R2t}-1)

= (_q-l R12 )'I/J;'l/Ji'I/J2'I/Jl - g'I/Ji I'l/J2( _q-l (R21 )-1)

= (_q-l R12 )'I/J;'l/J2( _q-l R12)'l/Ji'I/Jl - g( _q-l R12 )'I/J;I'l/Jl

- g'I/JU'l/J2(-q-1(R2t}-1)

= (_q-l R12)'I/J;'l/J2( _q-l R12)'l/Ji'I/Jl - g( _q-l R12 )'I/J;'l/J2P

- gP'I/J;'l/J2( _q-l (R21 )-l).

The proof for the case of if>* and if> is the same. The definitions of quantum Clifford algebras Cq (2n) and quantum Weyl

algebras Wq (2n) by the relations (4.2.1), (4.2.2), or (4.2.3), (4.2.4) and an analogue of Proposition 4.2.4 were discovered by physicists (See e.g. [PW) , [WZ), [K), [IP]).

One can construct spinor and oscillator representations of quantum groups Uq (o(2n» and Uq (.sp(2n» starting directly from the relations of quantum Clifford and Weyl algebra given in Proposition 4.2.1 or equiv­alently in Proposition 4.2.3. Since for the case of Uq (o(2n», the module Aq(Vq®Vq) is decomposed into three components as a module of Uq(gl(n», namely V~ ® v~, Aq(V~ ® V~) and Aq(V~ ® V~), we can construct 'irAq (Lvq) from the quadratic operators 'I/J*'I/J, 'I/J*'I/J* and 'I/J'I/J. Similarly in the case of

156 J. Ding and I. B. Frenkel

Uq(sp(2n», the module Sq(Vq ® V q) is decomposed into three components with respect to Uq(gl(n», namely V-t ® v~, Sq(V-t ® V-t) and Sq(V~ ® v~) and 1Tsq(Lvq ) is expressed by means of ¢J*¢J, ¢J*¢J* and ¢J¢J.

Finally we note that R-matrices in the tensor products Sq ® Sq and Aq ® Aq can be easily described using the realization of Aq and Sq as the exterior algebra A;(V-t) and the symmetric algebra S;(V-t), respectively.

5. Quantum Clifford algebra and quantum

Weyl algebra in canonical bases

Before we proceed to explicitly construct realizations of quantum Clifford and Weyl algebras, we would like to explain the implication of our invari­ant approach. We can start from a fundamental representation of Uq(o(N» and Uq(sp(2n» and find the R-matrix in this representation. Then we can define quantum Clifford and Weyl algebra by means of algebraic relations given in Sections 3 and construct 1T(.c), where 1T is the spinor or oscillator representation. Next we define End±(V) using a fixed ordered basis and according to results in Section 3, we obtain spinor and oscillator repre­sentations of the algebra Uq(.c±) and, via the isomorphism of Section 2, the representations of the quantum groups Uq(o(N» or Uq(sp(2n». Direct verification of Serre's relations is possible but a formidable problem (see e.g. [HD.

Thus the initial data of the construction is an N-dimensional vector space vq with an ordered basis and a certain solution of the Yang-Baxter equation R E End(Vq ® vq). Explicit formulas for such R are well known and will be recalled in the first subsection below.

Moreover the results of Section 4 provide another realization of quan­tum Clifford and Weyl algebras and therefore of spinor and oscillator rep­resentations using only the solution of the Yang Baxter equation in the fundamental representation V-t of the quantum group Uq(gl(n». This so­lution has a particularly simple form and contains essentially all the struc­tural information about the quantum Clifford and Weyl algebras and their representations. The results of the previous sections explain the conceptual mathematical reasons why and how all the explicit constructions work.

5.1. R-matrices for fundamental representations. In order to find an explicit expression for RV::' v::" we recall that P R is an intertwining operator for Uq(gl(n» on V-t ® V-t. Furthermore q-l and -q are known to be the eigenvalues of P R corresponding to two irreducible components Sq(V-t ® V-t) and Aq(V-t ® V-t), respectively. Also an overall constant of R can be found from (2.1.6). Choosing a canonical basis {ei}f=l for the space

Spinor and Oscillator Representations of Quantum Groups 157

v~, SO that el is the highest weight vector in V~ and ei+l = Fi(ei), i = 1,2, ... ,n - 1, where the Fi are generators of Uq(gl(n)), we obtain

Proposition 5.1.1. Projection of the universal R-matrix for Uq(gl(n» to the fundamental representation V~ has the following form in the basis ei,i = 1, ... ,n,

n n

R = L eii ® ejj + q L eii ® eii + (q - q-l) L eij ® eji' (5.1.1) i"lj i=l l::;i<j::;n

i,j=l

The matrix R also satisfies the following equalities

where h, t2 denote tmnsposition on the first or the second component of End (Cn ) ® End(Cn ), respectively.

This matrix was first given by Jimbo [J2]. The first identity in (5.1.2) implies that P R defines a representation of a Hecke algebra.

The R-matrices corresponding to Uq(o(2n + 1)), Uq(o(2n» and Uq(sp(2n» on vq ® vq can be derived similarly. From Section 2.4, we know that V q ® V q is decomposed into three irreducible components and the operator P R = P Rvqvq has the eigenvalues as in Propositions 2.4.2 and 2.4.3. The same argument as above yields the following

Proposition 5.1.2. Projection of the universal R-matrices for Uq(o(N)) and Uq(sp(2n» to the fundamental representation V q has the following form in the basis ei, i = 1, ... ,N

1Tvq ® 1TVq (vt12 ) = N

R = q L eii ® eii + e(N+1)/2,(N+1)/2 ® e(N+l)/2,(N+1)/2 i=l

N N

+ L eii ® ejj + q-l L ei'i' ® eii i,j=l i=l ii'j,j'

N 2n

(5.1.4)

+(q_q-l) L eij®eji-(q-q-l) 2:: qtl;'-tlj'cicjeij®ei'j', i,j=l i<j

i,j=l i<j

158 J. Ding and I. B. Frenkel

where the second term is for the case of Uq(o(2n + 1»; for Uq(o(2n + 1» and Uq(o(2n», if = N + 1 - i, Ci = 1, i = 1, ... , N , and for Uq(sp(2n», Ci = 1,i = 1, ... ,N/2, Ci = -l,i = N/2+ 1, ... ,N,· for Uq(o(2n+ 1», (U1 ... UN) = n-1/2,n-3/2, ... ,1/2,0, -1/2, .. . -n+1/2, for Uq(o(2n», (U1 ... UN) = n - 1,n - 2, ... ,1,0,0,-1, ... - n + 1, for Uq(sp(2n» (U1 ... l:'2n) = n, n - 1, ... ,1, -1, ... , -no f = 1 for Uq(o(2n» and f = -1 for Uq(sp(2n».

These matrices satisfy the following equalities:

where

R-1 = Rq-+q_l,

(PR - qI)(PR + q-1I)(PR - cqE-2nI) = 0,

PR - R-1 P = (q - q-1)(I - K),

2n K = L qU;-Ujcicjeij' ® ei'j·

i,j=l

(5.1.5)

These R-matrices were given by Jimbo [J3], Bozhanov [B] and Reshetikhin [R]. The properties (5.1.5) were stated in [FRT]. One can also check directly the idempotent property of K (2.4.3), (2.4.5) and the re­lation between K and F, Ff used in the proof of Proposition 3.1.5 and 3.3.5.

The selfduality of V q admits the following explicit form

Proposition 5.1.3. The matrices for the fundamental representations vq

of quantum groups Uq(o(N», Uq(sp(2n» satisfy the following symmetry conditions. Let Co = (CjeijDij') be a 2n x 2n matrix, and C = qUCo. Then

C1PR-1 PCl 1 = (R2d 1

C1C2PR-1p(CIC2)-1 = (R- 1)t1t2 ,

where C1 = C®I,C2 = I®C.

(5.1.6)

The matrix C is related to the invariant form in V q as in Section 2.1. The original invariant definition of the symplectic and orthogonal quantum groups [FRT] was given using the explicit expression of the matrix C as above. Conjugation by C combined with transposition induces the action of the antipode.

We note that (5.1.5) can be interpreted as the representation of Birman-Wenzl algebra [BW]. The structure of the Birman-Wenzl alge­bra is more complicated than structure of the Hecke algebra; however it admits a simple and transparent topological presentation [BW].

Spinor and Oscillator Representations of Quantum Groups 159

5.2. Realization of quantum Clifford algebra and quantum Weyl algebra. The explicit formulas for quantum Clifford and Weyl algebra have appeared previously in the physics literature (see e.g. [PW1, [WZ1, [K]). They can be derived using R-matrices either for quantum group Uq(gl(n)) or the quantum group Uq(o(2n)) and Uq(.sp(2n)). We will start with the former one.

Propositon 5.2.1. The quantum Clifford algebm Cq (2n) is isomorphic to an associative algebm with genemtors 'l/Jt, ... ,'l/Jn, 'l/Ji, ... ,'I/J~ satisfying the following relations:

if i > j, 'I/Ji'I/J; = -q'I/J;'l/Ji, if i > j, 'l/Ji'I/Ji = 'l/Ji'I/Ji = 0,

'l/Ji'I/J; = _q-1'I/J;'l/Ji, if i f= j, 'l/Ji'l/Ji + 'l/Ji'I/Ji = (q-2 - 1) E 'I/J;'l/Jj + 1.

i<j

(5.2.1)

The isomorphism is given by the identification of 'l/Ji with 'I/J( ei) and 'l/Ji with 'I/J*(ei ), where {ei} is a dual basis.

Proposition 5.2.2. The quantum Weyl algebm Wq (2n) is isomorphic to an associative algebm with genemtors ¢l1 ... ,¢In, ¢li ... ¢l~ satisfying the following relations

if i f= j

(5.2.2)

The isomorphism is given by the identification of ¢li with ¢l(ei) and ¢li with ¢l*(ei).

We can also express quantum Clifford and Weyl algebras using R­matrices for Uq (o(2n)) and Uq (.sp(2n)) respectively. Formulas (3.1.2) and (3.3.2) combined with (5.1.4) imply

Proposition 5.2.3. The quantum Clifford algebm Cq (2n) is isomorphic to an associative algebm with genemtors {Xd~!!1 satisfying the following relations:

-1 'f" XiXj = -q XjXi, Z Z > J,

Xi'Xi = -XiXi' + (q-2 - 1)'Ej>iqP;I-Pjl XjXjl + qP;I,for i > i'. (5.2.3)

160 J. Ding and I. B. Frenkel

Proposition 5.2.4 The quantum Weyl algebra Wq {2n) is isomorphic to an associative algebra with generators {Yi}~!!l satisfying the following rela­tions:

YiYj = qYjYi, if i > j, Yi'Yi = q2YiYi' + (q2 -1)'Ej>iqP;'-Pj'YjYj' + qP;I, for i > if. (5.2.4)

An isomorphism between the two realizations of the quantum Clifford algebras Cq (2n) is given by Xi = 1fJi, i = 1, ... ,n, Xi' = 1fJ;qP;' ,i = 1, ... ,n; and an isomorphism between the two realizations of the qunatum Weyl algebras Wq (2n) is given by Yi = <Pi, Yi' = -<piqP;' ,i = 1, ... ,n.

5.3. Realization of quantum algebras in classical algebras. Defor­mation of the category of highest weight modules implies that the (graded) dimensions of quantum Clifford and Weyl algebras for formal or generic values of q are exactly the same as their classical counterparts. Moreover since the quantum Clifford algebra is finite dimensional and semisimple, it has to be abstractly ismorphic to the classical Clifford algebra, a similar but a bit more subtle isomorphism should also exist for quantum Weyl al­gebras, which have infinite dimension. It turns out that the isomorphism between quantum and classical algebras can be constructed explicitly. As a corollary, this provides an explicit realization of quantum groups.

Let 1fJf, 1fJ~' i = 1, ... ,n be generators of the classical Clifford algebra satisfying the standard relations: {1fJf, 1fJ'j*} = 8ij . Let

Then, we have for i = 1, ... ,n

wi1fJfwi 1 = q6;j 1fJf wi1fJ;cwi 1 = q-6;j 1fJ;c

w~ = 1 + (q-2 - 1)1fJ;c1fJf .

Proposition 5.3.1. [HI Let

;Pi = II wj1fJf,;Pi = II Wj1fJ;c. j>i j>i

(5.3.1)

(5.3.2)

(5.3.3)

(5.3.4)

(5.3.5)

Then ;Pi';P; satisfy the relations (5.2.1) of the quantum Clifford algebra, where we set 1fJi = ;p and 1/Ji = ;Pi .

Spinor and Oscillator Representations of Quantum Groups 161

Proof. From (5.3.3), we have, for i > j,

A A -1 A A

"pi"pj = -q "pi"pi,

tPitP; = -qtP;tPi, (5.3.6) A A 1 A A

"pi"p; = -q- "p;"pi,

and, for i < j,

From the definition of the Clifford algebra, we have

From (5.3.5) and (5.3.4), we get

A A. A. A IT 2 "pi"pi +"pi"pi = Wj

j>i

= (1 + (q-2 - 1)"pf-i-1"pi'+l) IT w; j>i+1 (5.3.7)

Thus the commutation relations for tPi and tPi are the same as the ones for "pi and "pi for the quantum Clifford algebras in last subsection.

Classical Weyl algebra has a realization as an algebra of differential operators of n variables

W(2n) ~ C[X1 ••• X n , 8/ch1 , ••• ,8/8xn l.

Let Ti be a shift operator acting on polynomial functions which is defined by

(5.3.8)

We denote

1 2 Di = (2 ) (Ti - 1).

q - 1 Xi (5.3.9)

We enlarge W(2n) by adding formal power series in 8/8xi, i = 1, ... ,n, and we denote the enlarged algebra by W(2n). Clearly Di belongs to

162 J. Ding and I. B. Frenkel

W(2n) since we can expand (5.3.9) into a power series:

(5.3.10)

Proposition 5.3.6. Let

(5.3 .11)

be operators in W(2n). Then, J:, Ji satisfiy the relation (5.2.2) of the quantum Weyl algebra, where we set ¢* = J*, ¢i = Ji. Proof. We can easily check that if i > j,

if i < j,

JiJj = qJiJi, J:J; = q-lJ;J:, JiJ; = qJ;Ji,

Thus, we checked the commutation relations of the quantum Weyl algebra Wq (2n).

The realization of quantum Clifford and quantum Weyl algebras can be used to give an explicit construction of the quantum groups Uq (o(2n)) and Uq (sp(2n)) on the algebras of n anticommuting and commuting variables, respectively.

6. Further research directions

In this paper, we have taken only a few steps towards invariant approach to various representation theoretical constructions of quantum groups. One of the basic questions in this approach is the correct definition of the quantum analogue of Lie algebras. The quantum spinor and oscillator representa­tions yield naturally the linear space of operators that has the dimension

Spinor and Oscillator Representations of Quantum Groups 163

of the corresponding Lie algebras of classical types. The relations between these operators cast in the axiomatic form naturally lead to the notion of quantum Lie algebras and will be discussed in a subsequent paper.

Another immediate generalization of spinor and oscillator representa­tion involves quantum groups related to affine Lie algebras. The corre­sponding classical case was studied in detail in [FF]. In order to formulate the quantum analogue, one can use the relations between intertwining op­erators introduced and studied in [FR]. This problem will be the subject of another paper.

Our invariant approach may create the impression that essentially all familiar classical structures related to spinor and oscillator representations (as well as other representation theoretic constructions) can be routinely quantized without a significant new phenomena. However one should be prepared to unexpected miracles. Here we will mention only three possible sources for new structures.

The first new feature of the quantum case is the presence of the pa­rameter q. For generic or formal parameter, the representation theory is a deformation of its classical counterpart. But when q is special the picture can be dramatically different. For example if q2 = -1, one can deduce from the explicit presentation of quantum Clifford and Weyl algebras that they are isomorphic. If q is an m-th root of unity, then Cq (2n) and Wq (2n) are not simple anymore. They have ideals generated by 1/Jm, 1/Jm* ,m < n or ¢m, ¢*m respectively. The spinor and oscillator representations are not irreducible either. For m = n, the quantum Clifford and Weyl algebras co­incide with the algebras studied in [FV]. When q = ° or 00, one obtains the crystal structure of quantum Clifford and Weyl algebras and their repre­sentations which is reduced to an interesting combinatorical graph theory. In particular, one has the following crystal bases of Aq and Sq. For the cases of Aq , 1/Ji1 •• • 1/Ji k form a crystal basis, for q = 00, il < .. < ik; for q = 0, il > .. > ik. The crystal basis for Sq is given by ¢{~ ... ¢1:, for q = 00, il < .. < ik; for q = 0, il > .. > ik.

Another interesting new feature of quantum groups is their realization in terms of the affine analogues of the corresponding classical Lie alge­bras and their representations. Thus the representation of .c± in A and S can be given precisely by the "half" monodromy of trigometric Knizhnik­Zamolodchikov equation for the two points with values in A 18> V or S 18> V. The last new feature of the quantum case that we would like to mention is the existence of certain transformation between quantum deformations of correlation functions for representations of affine Lie algebras and their massive nonlinear deformations [S] that lies at the heart of complete inter­grability of two dimensional quantum field theories.

164 J. Ding and 1. B. Frenkel

References

[BW] J. Birman and H. Wenzl, Braids, link polynomial and a new algebra, Thms. of the American Math. Soc. 1313(1), (1989), 249-273.

[B] V.V. Bozhanov, Integrable quantum systems and classical Lie alge­bras, Comm. Math. Phys. 113 (1987), 471-503

[DF] J. Ding and LB. Frenkel, Isomorphism of two realizations of quantum affine algebra Uq(gl(n)), Comm. Math. Phys. 156 (1993), 277-300.

[Dl] V.G. Drinfeld, Hopf algebra and the quantum Yang-Baxter equation, Dokl. Akad. Nauk. SSSR 283 (1985), 1060-1064.

[D2] V.G. Drinfeld, Quantum groups, ICM Proceedings, New York, Berke­ley, 1986, 798-820.

[D3] V.G. Drinfeld, On almost cocommutative algebras, Leningrad Math. Jour., 1 (1990), 321-342.

[FV] L.D. Faddeev and A.Yu. Volkov, Hirota equation as an example of intergrable symplectic map, Hu- TFT-93-30 1993.

[FF] A.J. Feingold and LB. Frenkel, Classical affine algebras, Adv. in Math. 56 (1985), 117-172.

[FJ] LB. Frenkel and N. Jing, Vertex representations of quantum affine algebras, Pmc. Natl. Acad. Sci., USA 85 (1988),9373-9377.

[FR] I.B. Frenkel and N.Yu. Reshetikhin, Quantum affine algebras and holomorphic difference equation, Comm. Math. Phys. 146 (1992), 1-60.

[H] T. Hayashi, Q-analogue of Clifford and Weyl algebras - spinor and os­cillator representation of quantum enveloping algebras, Comm. Math. Phys. 127 (1990), 129-144.

[IP] A.P. Isaev and P.N. Pyatov, GLq(N)-covaiant quantum algebras and covariant differential calculus, Physics Lett. A 179 (1993), 81-90.

[Jl) M. Jimbo, A q-difference analogue of U(g) and Yang-Baxter equation, Lett. Math. Phys. 10 (1985), 63-69.

[J2) M. Jimbo, A q-analogue of Uq (gl(n+l)), Hecke algebra and the Yang­Baxter equation, Lett. Math. Phys. 11 (1986),247-252.

[J3] M. Jimbo, Quantum R-matrix for the generalized Toda systems, Comm. Math. Phys. 102 (1986),537-548.

[J4] M. Jimbo, Introduction to the Yang-Baxter equation, Int. Jour.of Mod. Phys. A 4 (15), (1990), 3758-3777.

Spinor and Oscillator Representations of Quantum Groups 165

[K] P.P. Kulish, Finite-dimensional Zamolodchikov-Faddeev algebra and q-oscillators, Physics Lett. A 161 (1991), 50-52.

[L] G. Lusztig, Quantum deformations of certain simple modules over enveloping algebras, Adv. Math. 70 (1988), 237-249.

[PW] W. Pusz and S. Woronowicz, Twisted second quantization, Rep. Math. Phys. 27 (1989), 231-257.

[R] N.Yu. Reshetikhin, Quantized universal enveloping algebras, the Yang-Baxter equation and invariants of links I, II, 1987-1988, LOMI, Preprint E-4-87, E-17-87, L: LOMI.

[FRT] N.Yu. Reshetikhin, L.A. Takhtajan, and L.n. Faddeev, Quantization of Lie groups and Lie algebras, Leningrad Math. J. 1 (1990), 193-225.

[RS] N.Yu. Reshetikhin and M.A. Semenov-Tian-Shansky, Central exten­sions of quantum current groups, Lett. Math. Phys. 19 (1990), 133-142.

[8] F.A. Smirnov, Introduction to quantum groups and integrable mas­sive models of quantum field theory, Nankai Lectures on Mathematical Physics, Mo-Lin Ge, Bao-Heng Zhao(eds.), World Scientific, 1990.

[VO] E.B. Vinberg and A.1. Onishchik, Lie groups and algebraic groups, Springer-Verlag, New York 1990.

[WZ] J. Wess and B. Zumino, Covariant differential calculus on the quan­tum hyperplane, Nucl. Phys. B (Proc. Supp.) 18B (1990), 302-312.

Department of Mathematics, Yale University, New Haven, CT 06520, USA.

Received May 18, 1994

Familles coherentes sur les groupes de Lie semi-simples

et restriction aux sous-groupes compacts maximaux

Michel DuRo and Michele Vergne

Summary.

Let G be a semi-simple simply connected Lie group and K be a sub­group, maximal among the subgroups compact modulo the center. In­variant eigendistributions on G can be restricted to K. It is an irritating fact that, without supplementary assumptions, this restriction does not determine the distribution. However, we prove here that this is true if, instead of a single invariant eigendistribution, one considers a family of invariant eigendistributions, coherent in the sense of W. Schmid. We also study non connected semi-simple groups: we define a notion of coherent family of invariant eigendistributions and prove a unicity theorem in this setting.

Table des matieres.

Introduction.

1 Le cas des groupes connexes. 1.1 Familles coMrentes sur Ie groupe. 1.2 Familles coMrentes de fonctions generalisees sur l'algebre de Lie. 1.3 La famille principale B;"inc' 1.4 Familles s-coMrentes et descente. 1.5 Demonstration des tMoremes d'unicite.

2 Groupes non connexes. 2.1 Familles coMrentes de fonctions generalisees: enonce du tMoreme. 2.2 Un exemple de famille coMrente : la formule de Kostant. 2.3 Familles s-coherentes et descente. 2.4 Demonstration des tMoremes d'unicite.

References.

168 M. Du/lo and M. Vergne

Introduction.

Soit L un groupe de Lie reel semi-simple admettant un nombre fini de composantes connexes. NollS notons G la composante neutre de L et 9

son algebre de Lie. Soient U(gc) l'algebre enveloppante du complexifle gc

de g, identifiee comme d'habitude a. l'algebre des operateurs differentiels sur L invariants par les translations a. gauche, et Z(gc) Ie centre de U(gc). Soit V un ouvert G-invariant de L. Nous etudions des fonctions generali­sees G-invariantes I E C-OO(V)G qui sont vecteurs propres pour l'action de Z(gc).

Soient K un sOllS-groupe de G, compact maximal modulo Ie centre et LK Ie normalisateur de K dans L. NollS considerons des ouverts G­invariants elliptiques V de L. La definition d'ouvert elliptique est redonnee plus bas. Rappelons simplement qu'un ouvert elliptique est determine par son intersection avec LK et que les ouverts elliptiques recouvrent L.

On sait qu'une fonction generalisee I comme ci-dessus admet une restriction IILK a. LK n V qui est une fonction generalisee sur LK n V. Par exemple, si I est Ie caractere d'une representation unitaire irreductible T de L dans un espace de Hilbert (ou plus generalement d'un (g, LK )­

module simple), la fonction generalisee IILK decrit la decomposition de la restriction TILK de T en composantes isotypiques sous l'action du groupe LK. Dans un certain nombre de situations, on connait Ie caract ere infinitesimal de I et la restriction IILK' II est donc utile de savoir quand ces deux donnees determinent I, ce qui n'est pas Ie cas sans conditions supplementaires. Le prototype de ces tMoremes d'unicite est do a. Harish­Chandra [13] et joue un rOle important dans sa description de la serie discrete du groupe G, ou dans la demonstration de la formule de Rossmann pour les caracteres des series discretes. Le tMoreme d'unicite d'Harish­Chandra requiert notamment des hypotheses de croissance sur I.

Les tMoremes d'unicite que nous demontrons ici sont voisins d'un result at de David Vogan [22] sur les (g, K)-modules. lis ne demandent pas de conditions de croissance, mais portent sur toute une famille de fonctions generalisees, coMrente au sens de W. Schmid [17]. Soit june algebre de Cartan de Dc. Soit j* Ie dual de j. On sait que chaque A E j* definit un caractere X,\ de Z(gc). On note P c j* Ie reseau des poids.

Supposons pour commencer V contenu dans G. Vne famille coMrente de fonctions generalisees dans Vest une application A 1-+ F(A) definie sur un translate P+AO de P, ou chaque F(A) est une fonction generalisee G-in­variante sur V de caractere infinitesimal X,\, verifiant une certaine propriete de moyenne (voir la formule (3) ci-dessous).

Familles coherentes 169

Theoreme 1 Soit V un ouvert elliptique G-invariant de G. On suppose que toute representation de dimension finie de 9 s 'integre en une representation de G. Soit,x 1--+ F{,x) une famille coMrente de fonctions ge­neralisees sur V. Supposons que l'on ait F{,x)IK = 0 pour tout ,x E ,xo + P. Alors F{,x) = 0 pour tout ,x E ,xo + P.

Le tMoreme 1 est enonce plus precisement dans Ie tMoreme 6 du paragraphe 1.1.

Lorsque V = G, on peut deduire Ie tMoreme 1 du resultat de D. Vogan sur les familIes coMrentes de (9, K)-modules ([22), lemma 8.2) mentionne ci-dessus. Inversement, Ie thooreme 1 en donne une autre demonstration.

Soit maintenant Y une composante connexe de L, et supposons V contenu dans Y. On associe a Y un automorphisme a de ; et de son systeme de racines. L'automorphisme a est trivial si et seulement si Ad{y) est un automorphisme interieur de 9 ® C pour tout Y E Y. Notons ;*(a) Ie sous­espace des points fixes de a dans j* et P{a) = P n ;*(a). Nous definissons (definition 45) une notion de familIes coMrentes de fonctions generalisees G-invariantes sur V. Une famille coMrente est une application ,x 1--+ F{,x) definie sur un translate P(a) + ,xo de P(a) dans ;*(a), ou chaque F{,x) est une fonction generalisee G-invariante sur V de caractere infinitesimal x-\, verifiant une certaine propriete de moyenne. Nous demontrons pour ces families une generalisation du tMoreme 1 (tMoreme 46 du paragraphe 2.1).

Les tMoremes 1 et 46 sont obtenus par la methode de descente, habituelle dans les travaux d'Harish-Chandra sur les distributions invariantes. Ceci amene a definir sur 9 et certaines sous-algebres reductives de 9 une notion de familIes coMrentes de fonctions generalisees, et a etablir pour celles-ci des tMoremes d'unicite (theoremes 16 et 30). Cette notion est interessante independamment de l'application faite ici. Nous montrerons ailIeurs comment fabriquer de telles families coherentes sur 9 en integrant des formes differentielles equivariantes sur des espaces homogEmes de G.

Dans un article ulterieur, nous nous servirons des theoremes d'unicite etablis ici pour demontrer la formule pour les caracteres des modules de Zuckerman annoncee dans [21]. C'est la motivation initiale de ce travail.

En esperant etre plus clairs, et bien que cela entraine quelques repetitions, nous traitons separement Ie cas des groupes connexes dans une premiere partie, et sa generalisation aux groupes non connexes dans une seconde.

Les resultats de cet article ont ete annonces dans [8].

Nous remercions David Vogan pour d'utiles commentaires.

170 M. Du.flo and M. Vergne

Notations.

Dans tout l'article, les notations suivantes seront employees. Si Vest un espace vectoriel, son dual est note V*, son algebre symetrique S(V). Si V et V' sont des espaces vectoriels, on note L(V, V') l'espace des applications lineaires de V dans V'. Si Vest reel, on note Vc son complexifie.

Si un groupe A opere dans un ensemble X, on note X A l'ensemble des elements de X fixes par A. Si a E A, on note X(a) l'ensemble des elements de X fixes par a. Si x E X, on note A(x) Ie groupe des elements fixant x.

Si M est une variete differentiable reelle, on note COO(M) l'espace des fonctions differentiables a valeurs complexes sur M, C:;C>(M) Ie sous­espace des fonctions a support compact et C-OO(M) l'espace des fonctions generalisees, c'est-a-dire Ie dual au sens de L. Schwartz de l'espace des densites Coo a support compact sur M.

Dans tout cet article, D est une algebre de Lie semi-simple reelle, Gc Ie groupe de Lie complexe connexe et simplement connexe d'algebre de Lie Dc et G un groupe de Lie connexe d'algebre de Lie D. On suppose que l'injection de D dans Dc s'integre en un homomorphisme K de G dans Gc. II revient au meme de dire que toute representation de dimension finie de D s'integre en une representation de dimension finie de G. On note r Ie noyau de K. C'est un sous-groupe du centre de G. On fixe un sous-groupe K de G tel que Klr soit un sous-groupe compact maximal de G/r.

1 Le cas des groupes connexes.

1.1 FamilIes coherentes sur Ie groupe.

On note G~ini l'ensemble des classes d'equivalence de representations irre­ductibles de dimension finie holomorphes de Gc et R( Gc) Ie groupe libre sur Z de base indexee par G~ini. Le produit tensoriel de representations induit sur R( Gc) une structure d'anneau commutatif avec unite. Si Lest une variete algebrique affine complexe, on note F(L) l'algebre des fonctions regulieres sur L. On considere Gc comme un groupe algebrique affine pour lequel F( Gc) est l'algebre engendree par les coefficients des representations T E G~ini. On note F(Gc)Gc la sous-algebre de F(Gc) formee des elements Gc-invariants par l'action adjointe. On associe a un element T = E nm (ni E Z) de R(Gc) la fonction centrale 9 1-+ TrT(g) = Eni TrTi(g). On dit que c'est la trace de T. Elle appartient a F(Gc)Gc.

Nous dirons qu'une representation de D dans un espace vectoriel complexe Vest un (D, K)-module si elle s'integre en une representation localement de dimension finie du groupe K dont la restriction a rest somme

Familles coherentes 171

directe de caracteres unitaires. On note R-OO(G) Ie groupe libre sur z ayant pour base l'ensemble des classes d'equivalence de (9, K)-modules irreduc­tibles. Un element de R-OO(G) sera appele un (9, K)-module admissible virtuel. Si V est un (9, K)-module de longueur finie, on lui associe un element [V] de R-OO(G) par addition des quotients simples d'une suite de composition de V. La composition avec /'i, induit une bijection entre les classes de representation de dimension finie de G et les representations de dimension finie holomorphes de Ge. Par produit tensoriel, l'ensemble R-OO(G) est un module sur R(Gc).

Si Vest un (9, K)-module irreductible, la representation T de 9 dans Vest la differentielle d'une representation (notee encore T) continue de G dans un complete hilbertien de V. La representation T est it trace et la trace de Test une fonction generalisee G-invariante sur G, notee TrT, qui ne depend pas de la completion choisie. On associe it un element T = L: niT; (ni E Z) de R-OO(G) la fonction generalisee TrT(g) = L:niTrTi(g). On dit que TrT est un caractere virtuel de G. Si T E R(Gc) et T' E R-OO(G), on a Tr(T ® r) = TrTTrT'.

Si ! est une algebre de Lie reductive complexe et c une sous-algebre de Cartan de !, on note ~(!, c) C c* Ie systeme des racines.

Si c et c' sont deux sous-algebres de Cartan de !, on note W.(c, c') l'espace des applications lineaires de c dans c' qui sont restriction it c d'un automorphisme interieur de!. En particulier, W.(c, c) est Ie groupe de Weyl, et W.(c, c') est un espace homogene principal it droite sous Ie groupe W.(c, c). Lorsque ! = ge, nous ecrirons simplement W(c, c') = W.(c, c').

On fixe une algebre de Cartan ; de ge et un systeme de racines positives ~ + c ~(ge, ;). On note ~ = ~(ge,;) C ;*, p la demi-somme des racines de ~+, W = We;,;) et feW) la signature d'un element W E W.

Soit J Ie sous-groupe de Cartan de Ge d'algebre de Lie;. On note P c ;* Ie reseau des poids. Done P est Ie reseau forme des differentielles des caracteres rationneis de J. Si A E P, on note e A Ie caractere correspondant de J. On note z(;*) l'algebre du groupe ;*. On note encore eA l'element de zO*) correspondant it A E ;*. On note R(J) Ie groupe libre sur Z de base P. Un element de R(J) est noteL:A nAeA, et R(J) est une sous-algebre de z(;*). Si Ao E ;*, Ie sous-groupe R(J)eAO C z(;*) est un module libre sur R(J) de rang 1. Par restriction it J un element A de R( Gd fournit un element a = AIJ de R(J). Ceci permet de considerer R(J) comme une algebre sur R(Gc), et tout R(J)-module (et en particulier Ie module R(J)eAO ) comme un R( Gc)-module.

Soit P+ l'ensemble des poids dominants. Si A E P+, on note yeA) Ia representation irreductible de dimension finie de Gc de poids dominant A.

172 M. Duflo and M. Veryne

Si ( est une algebre de Lie, on note U«() son algebre enveloppante et Z«() Ie centre de U(t). Soit h j : Z(gc) -+ S(;) l'homomorphisme d'Harish­Chandra. Pour >. E ;*, on note XA Ie caractere de Z(gc) donne par XA(U) = h;(u)(>.). En particulier, si >. E P+, Ie caractere infinitesimal de la representation V(>') est egal 8. XHp' On note R-OO(G)A Ie sous-module engendre par les (g, K)-modules irroouctibles de caractere infinitesimal XA'

La notion de familIes coherentes est essentiellement due 8. W. Schmid [17]. Nous renvoyons 8. [23], [24] et [27] pour plus de details. Nous allons en utiliser plusieurs variantes. Rappelons tout d'abord la definition de familIes coherentes de (g, K)-modules.

Definition 2 Soit >'0 E ;*. On appeUe famille coMrente de (g, K)-modules basee sur P + >'0 une application C : R(J)eAO -+ R-OO(G) telle que

(1) l'application C est un morphisme de R(Gc)-modules, (2) pour tout>. E P + >'0, on a C(eA) E R-OO(Gh.

Par exemple, il existe une familIe coherente Bf;n basee sur P et une seule telle que l'on ait Bf;n(eA) = V("\ - p) pour tout ..\ E P dominant regulier. On a Bf;n(eA) = 0 si ..\ n'est pas regulier, et Bf;n(eWA ) = €(w)Bf;n(e A) pour tout ..\ E P et w E W. Ceci decoule immooiatement de la formule d'Hermann Weyl pour Ie caractere de V("\).

Soit ;' une autre sous-algebre de Cartan de gc, et soit ,6.'+ un systeme de racines positif pour ,6.(ge,;'). II existe un unique element u E W(;,;,) qui transforme ,6.+ en ,6.'+. On se sert de u pour identifier R( J) et R( J'). On voit done que la notion de familIe coherente est independante du choix du couple (;,,6.+).

Soit V un ouvert G-invariant de G. Soit C-OO(V)G l'espace des fonctions generalisees G-invariantes sur V. On identifie U(gc) 8. l'algebre des operateurs differentiels sur G invariants par les translations 8. gauche. Si >. E ;*, on note C-OO(V)r Ie sous espace de C-OO(V)G forme des fonctions f verifiant U· f = XA(u)f pour tout u E Z(gc). Rappelons l'homomorphisme '" : G -+ Ge. La composition avec", est un isomorphisme de F( Gc) sur une sous-algebre de fonctions analytiques sur G. Par multiplication, l'espace C-OO(V)G est un module sur F(Gc)Gc . D'autre part, la restriction it J donne un morphisme de l'algebre F(Gc)Gc dans F(J). On peut done considerer tout F(J)-module comme un F(Gc)Gc-module. Soit..\o E t· Nous identifions F(J) 8. la sous-algebre de C(;*) formee des combinaisons lineaires it coefficients complexes 2:AEPcAeA• Le F(J)-module F(J)eAO C

C(;*) est donc aussi un module sur F(Gc)Gc.

Familles coherentes 173

Definition 3 Soit >'0 E ;*. Soit V un ouvert G-invariant de G. On appeUe lamille cohCrente de lonctions generalisees sur V basee sur P + >'0 une application \}I : F(J)e>-.o - C-OO(V)G telle que

(1) l'application \}I est un morphisme de F(Gc)Gc-modules, (2) pour tout>. E P + >'0 on a \}I(eA) E C-OO(V)f.

On note <I>(V, P + >'0) l'espace de ces familles coMrentes.

Soit V(J) l'algebre des operateurs differentiels de F(J). EUe est engendree par S(;) et F(J). Soient V(Gc) l'algebre des operateurs differentiels de F(Gc) et VGc sa sous-algebre engendree par F(Gc)Gc et Z(gc). D'apres Harish-Chandra [12], il existe un homomorphisme d'algebres, que nous notons encore hj, de VGc dans V(J), qui cOIncide avec hj sur Z(gc), et avec l'application de restriction sur F( Gc)Gc . Le F( J)-module F( J)e AO a une unique structure de V( J)-module teUe que S(;) opere sur eA (>. E P + >'0) par Ie caractere >.. D'autre part, VGc opere par des operateurs differentiels a coefficients analytiques dans C-oo (V)G. Les relations (1) et (2) s'ecrivent

(1) U' \}I(I/» = \}I(hj(u)· 1/»

pour tout U E VGc et I/> E F(J)eAO '

Soit /1 E P. On note II' l'element de F(Gc)Gc tel que

(2)

Les II' engendrent F( Gc)Gc .

Rappelons qu'un element >. E ;* est dit regulier si w>. -I- >. pour tout wE W, w -I- 1.

Lemme 4 Supposons >'0 regulier dans t. On a F(J)eAO = hj(VGc)eAo '

Demonstration. Soit >. E P + >'0' Montrons que eA appartient a hj(VGC)eAO' Notons (.,.) la forme bilineaire sur j* deduite de la forme de Killing. On choisit /1 E P tel que I'on ait (/1+>'0, /1+>'0) -I- (W/1+>'o, W/1+>'o) et (/1+ >'0 +w(>. - >'0 - /1), /1+ >'o+w(>. - >'0 - /1)) -I- (oX, >.) pour tout W E W, W -I- 1. Un tel /1 existe. En effet, comme P est dense pour la topologie de Zariski, il suffit de montrer que pour W -I- 1, chacune de ces deux relations a au moins une solution /1 E ;*. La premiere s'ecrit (/1, >'0 - w>'o) -I- 0, et eUe a une solution parce que >'0 - w>'o est non nul. Pour la seconde, on choisit un vecteur propre /10 de w pour une valeur propre differente de 1.

174 M. Duflo and M. Ve1IJTle

Alors la fonction t 1-+ (tJ.to + AO +W(A - AO - tJ.to) , tJ.to + AO +W(A - AO - tJ.to)) sur C est de degre 2, et donc non constante.

On considere hj(j,..)e>.o = EWEWew,..+>.o' Soit w E S(;) Ie laplacien defini par (.,.). L'operateur rrWEW.W~l(W - (wJ.t + AO,WJ.t + AO)) annule tous les ew,..+>.o correspondant a W =1= 1, mais it n'annule pas e,..+>.o' Comme I'operateur west dans I'image de h;, on a montre que I'on a e,..+>.o E h;('DGc)e>.o.

De la meme maniere, on demontre que I'on a e>. E hj(VGc)e,..+>.o. •

Soit AO E j* un element regulier. Il resulte du lemme 4 qu'une famille coherente IJ! E <I>(V, P + AO) est determinee par la fonction generalisee lJ!(e>.o) E C-OO(V)Xo' Reciproquement, si Vest completement invariant au sens d'Harish-Chandra [12], et si f E C-OO(V)Xo' it existe une famille coherente IJ! E <I>(V, P + AO) telle que lJ!(e>.o) = f. C'est un resultat plus profond (voir [17] et [27]). Nous demontrerons un result at analogue sur l'algebre de Lie (proposition 10).

Soit IJ! E <I>(V, P + AO) comme dans la definition 3. Pour A E P + AO posons F(A) = lJ!(e>.). La fonction F verifie les conditions

(3) E F(wJ.t + A) = f,..F(A) wEW

pour tout A E P + AO et J.t E P et

(4)

pour tout A E P + AO. On voit que la definition 3 est equivalente a la definition suivante.

Definition 5 Soit AO E ;*. Soit V un ouvert G-invariant de G. On appelle famille coherente de fonctions generalisees sur V basee sur P + AO une application F : P + AO -+ C-OO(V)G verifiant les proprietes (3) et (4).

On note .r(V, P + AO) l'espace de ces familIes coherentes.

Si C est une famille coherente de (g, K)-modules basee sur P + Ao, la famille F(A) = TrC(e>.) est une famille coherente de fonctions generalisees sur G basee sur P + AO'

On dit qu'un element s E G est elliptique si Ad(s) est contenu dans un sous-groupe compact d'endomorphismes de g, c'est-a.-dire si s est contenu dans un conjugue du groupe K. On note Gell l'ensemble des elements elliptiques de G. On dit qu'un element X E 9 est infinitesimalement elliptique si ad(X) est semi-simple avec des valeurs propres imaginaires

Families coherentes 175

pures, hyperbolique si ad(X) est semi-simple avec des valeurs propres reelles, et nilpotent si ad(X) est nilpotent. On note gell Ie sous-ensemble des elements infinitesimalement elliptiques de g, 9hyp celui des elements hyperboliques et gnil celui des elements nilpotents.

Un element 9 E G s'ocrit de maniere unique sous la forme

(5)

OU 8(g) E Gel" H(g) E ghyp et N(g) E gnil commutent deux It deux. Un ouvert G-invariant V de G est dit elliptique si, pour tout 9 E G, on a 8(g) E V si et seulement si 9 E V (voir [71, definition 35). Les ouverts elliptiques sont completement invariants.

Soit V un ouvert de G. Si I E C-OO(V) est une fonction propre pour l'operateur de Casimir, on sait que I admet une restriction 11K It V n K. En effet, soient t l'algebre de Lie de K et p l'orthogonal de t dans 9 pour la forme de Killing. Done 9 = t E9 pest une decomposition de Cartan de g. Identifions l'espace cotangent en un point 9 E V It g* grace it la translation it gauche par g, puis It 9 grace it la forme de Killing. Le front d'onde de I en chaque point 9 E Vest contenu dans l'ensemble N des elements de 9 isotropes pour la forme de Killing. Comme l'intersection de N avec l'orthogonal p de t est reduit it 0, Ie front d'onde est transverse It la sous-variete V n K de V.

Nous pouvons enoncer Ie theoreme principal de cette partie. Sa demonstration fait l'objet de la suite du chapitre.

Theoreme 6 Soit V un ouvert G-invariant elliptique de G. Soit >'0 E ;*. Soit F E :F(V, P + >'0)' On suppose que pour tout>. E P + >'0 on a F(>')IK = O. Alors F = O.

1.2 FamilIes coherentes de fonctions generalisees sur l'algebre de Lie.

Nous donnons deux versions infinitesimales de la notion de famille coherente. Soient U un ouvert G-invariant de 9 et C-OO(U)G l'espace des fonctions generalisees G-invariantes sur U. On note Chol(gc)Gc l'espace des fonctions holomorphes G-invariantes sur gc. L'espace C-OO(U)G est un module pour S(gc)GC operant par differentiation et un module pour Chol(gc)GC operant par multiplication.

Soit Chol(;) l'espace des fonctions holomorphes sur;. Par restriction it;, l'anneau Chol(gc)Gc s'identifie it un sous-anneau de Chol(;). En particulier Chol(;) est un Chol(gc)Gc-module. L'espace S(;) opere sur Chol(;) par differentiation. Soit c; : S(gc)Gc ~ S(;) l'application de restriction it

176 M. Duflo and M. Ve1Yne

;* C g~. La definition suivante est un analogue infinitesimal de la definition 3.

Definition 7 Soit 9 : Chol (;) -+ C-OO(U)G. On dit que 9 est une famille coherente de fonctions generalisees sur U si

(1) 9 est un morphisme de ChOI(gc)Gc-modules, (2) pour tout U E S(gc)GC et tout c/J E Chol(;) on a u·9(c/J) = 9(cJ(u),c/J).

On note «P(U) l'espace vectoriel des familles coMrentes de fonctions gene­ralisees sur U.

Exemple 8 On note Bjin : Chol(;) -+ Chol(gc)Gc l'application telle que, pour c/J E Chol(;) et X E ;, on ait

L'application Bjin est une famille coMrente de fonctions generalisees sur g.

Nous verrons un autre exemple de famille coMrente de fonctions generalisees sur 9 dans Ie paragraphe 1.3. Dans un prochain article, nous construirons des familles coMrentes de fonctions generalisees sur 9 en integrant des formes differentielles G-equivariantes sur des espaces homogEmes de G.

Considerons l'algebre d'operateurs differentiels lineaires holomorphes Vi sur; engendree par Chol(;)W et S(;)W et la sous-algebre Vjl engendree par S(;)W et S(;*)W. D'apres 126] (pour presque tous les groupes) et 116] (en general) vr' est egale a l'algebre des operateurs differentiels polynomiaux W-invariants sur;. On considere de meme l'algebre d'operateurs differentieis lineaires holomorphes VIJC sur ge engendree par Chol(gc)Gc et S(gc)GC et la sous-algebre V~ engendree par S(g~)Gc et S(gc)Gc. II existe d'apres Harish-Chandra [10] un homomorphisme d'algebres encore note ci de VIJC dans VI dont Ia restriction a Chol(gc)Gc est l'application de restriction, et Ia restriction a S(gc)Gc l'application ci'

Rappeions Ie lien entre ci et Ia composante radiale des operateurs differentiels de VIJC' On pose

D = II a E S(;*). oELl.+

Si I est une aigebre de Lie reductive, on note Ireg l'ouvert de I forme des elements reguliers, c'est-a-dire dont Ie centralisateur est une algebre de Cartan de I. Si d est un operateur differentiel sur ;, on considere D-1d D

Families cohtrentes 177

comme un operateur differentiel sur l'ouvert j n gc,regde j. Pour u E Vgc, on pose c;(u) = D-1c;(u)D. Soit a une algebre de Cartan de g. On definit de meme 1'0perateur differentiel holomorphe c,,(u) dans 1'0uvert des elements reguliers de Oc. Soit U un ouvert G-invariant de g. Toute fonction generalisee f E C-OO(U)G admet une restriction flor.gn" qui est une fonction generalisee dans greg nan U. Soit u E Vgc. On considere u comme un operateur differentiel a coefficients analytiques sur 9 et c,,(u) comme un operateur differentiel a coefficients analytiques dans greg n a. On a (Harish-Chandra [10])

(6)

Grace a I'homomorphisme surjectif c;, on considere Chol(j) comme un Vgc-module. On voit donc qu'une famille coherente 9 E <I>(U) est un morphisme de Vgc-modules de Chol(j) dans C-OO(U)G. 11 en resulte que si ¢J E Chol(;) engendre Ie Vi-module Chol(j) , la famille 9 E <I>(U) est determinee par la fonction generalisee 9(¢J) E C-OO(U).

Soit A E ;*. On note eA E Chol(j) la fonction X 1--+ eA(X).

Lemme 9 Soit A E j* un element regulier. Alors eA engendre Ie Vj -module Chol(j).

Demonstration. Nous noterons 1{ Ie sous-espace de S(j*) forme des polynomes harmoniques, c'est-a-dire annules par les operateurs differentiels definis par les elements sans terme constant de S(j)w. Rappelons ([6], chap. 5, no 5.2, tho 2 et [3], prop. 3.3) que la multiplication induit un isomorphisme de S(;*)W ® 1{ sur S(;*) et de Chol(;)W ® 1{ sur Chol(j). Soit P E 1{. 11 nous suffit de montrer que l'on peut obtenir p a partir de eA

en appliquant un operateur differentiel de V;. On considere la fonction invariante X 1--+ q(X) = EWEWp(wX)e-A(wX). On a

qeA = p + E wpeA- WA . wEW,w;il

Soit, comme dans Ie lemme 4, w Ie laplacien sur ; associe a la forme de Killing (., .). Pour tout JL E ;* 1'0perateur differentiel e-I-'(w - (JL, JL))el-' est sans terme constant. Donc, si P est un polynome de degre < n, on a (w - (fL, JL))n(Pel-') = O. Choisissons n strictement superieur au degre de p. 11 en resulte que 1'0n a

(w - (A - WA,A - WA)t· (wpeA- WA ) = 0

178 M. Dufto and M. Vergne

et done

( II (w - (>. - w>., >. - w>.)t) . qeA = wEW,w;oH

= ( II (->. + w>.,>. - w>.)n) p = cp wEW,w;ofl

avec une eonstante c non nulle. •

Soit >. E ;*. On note C-OO(U)~ l'espace des fonctions generalisees G­invariantes f sur U verifiant u . f = u(>.)f pour tout u E S(gc)Gc . Soit 9 E <J.>(U) une famille eoherente. La condition (2) de la definition 7 implique que l'on a 9(eA) E C-OO(U)~ pour tout), E ;*. II resulte du lemme 9 que si ), est regulier, la fonction 9(eA) determine 9.

La demonstration de la reciproque utilise Ie theoreme de regularite d'Harish-Chandra sur les fonetions generalisees invariantes que nous eommenc;ons par rappeler. Nous Ie faisons pour une algebre reductive reelle ~, car nous en aurons besoin plus bas. Soit ID~lla fonction sur ~ definie de la maniere suivante. Soit X E ~reg. Le centralisateur de X dans ~ est une algebre de Cartan a de~. On pose ID~I(X) = I det ~/a(X)ll/2. Si X n'est pas regulier, on pose ID~I(X) = o. Harish-Chandra a demontre que la fonction ID~I-l est loealement sommable sur~. Lorsque ~ = g, nous notons ID~I = IDI. Soit X E greg et soit a Ie centralisateur de X dans g.

On a IDI(X) = ID(wX)1 si w E W(ac,;). Soit U un ouvert completement invariant de ~, au sens d'Harish-Chandra [11]. Notons C-OO(U)lj;n l'espace des fonetions generalisees H-invariantes sur U qui sont annuIees par un ideal de codimension finie de S(~c)H. Soit f E C-OO(U)lj;n- Le theoreme de regularite d'Harish-Chandra [11] (voir [20] ou [25]) dit que fest une fonetion localement sommable, analytique dans ~reg n U.

Proposition 10 Soit U un ouverl completement invariant de g. Soit), un eiCment regulier de ;*. Soit f E C-OO(U)~. Alors il existe une et une seule famille coherente 9 E <J.>(U) telle que l'on ait 8(eA) = f.

Demonstration. L'unicite est deja etablie, et n'utilise d'ailleurs pas que U soit completement invariant. Demontrons l'existenee. Soient a une algebre de Cartan de g, et V une eomposante connexe de una n greg.

Comme f appartient a C-OO(U)~, la restriction de f a Vest donnee par la formule

(7) eA(W-1X)

f(X) = L cv(w) D(w-1X) WEW(j,ac)

Familles coherentes 179

ou les cv(w) sont des constantes (dependant de V, mais pas de X E V). Soit c/J E Chol(j). II existe une unique fonction analytique de£inie dans

U n greg telle que, pour toute algebre de Cartan a de g, toute composante connexe V de Una n greg, et X E V, on ait

(8) c/J(w-1X)

9'(c/J)(X) = E cv(w) D(w-1X)' wEW(J,ac)

II est clair que 9' est une famille coherente dans Un greg telle que 9'(e.\) soit la restriction de f a U n greg'

Notons 9(c/J) la fonction generalisee dans U definie par 9'(c/J). II s'agit de montrer que 9 est une famille coherente. La condition (1) de la definition 7 est vraie par construction. Pour verifier la condition (2), on remarque qu'il suffit de la demontrer pour des applications c/J de la forme pe.\ avec p E S(j*) et d'employer un argument de continuite.

Nous utiliserons les deux lemmes suivants.

Lemme 11 Soit A E j* un element regulier. On a Vj'e.\ = S(j*)e.\.

Demonstration. On emploie les notations de la demonstration du lemme 4. Soient J.L E j* tel que (J.L, A - w>.) =I- 0 pour tout w E W, w =I- 1, et n un entier 2: O. II suffit de montrer que J.Lne.\ s'obtient a partir de e.\ en appliquant une combinaison lineaire d'operateurs de la forme w4 EWEW(WJ.L)b ou a et b sont des entiers 2: O. Pour tout tEe calculons nWEW,w;iol(W - (tWJ.L + A,tWJ.L + A»(EwEWetWI')e.\. On trouve (2t)#W-l nWEw,w;iol (J.L, A - wA)etl'+.\. Le resultat s'obtient en developpant cette egaliM suivant les puissances de t. •

Lemme 12 Soit V un espace vectoriel complexe. Soit m l'ideal de S(V) engendre par V. On considere le coproduit qui envoie S(V) dans S(V) ® S(V). Soit a un ideal de S(V). Pour tout entier r > 0 l'image de ar est contenue dans S(V) ® a + mr ® S(V).

Demonstration. C'est clair. • On applique Ie lemme 12 a V = gc. Soit P E S(g~). Soit a l'ideal

de S(gc) engendre par les elements de S(gc)Gc nuls en >.. La fonction fest annulee par a, et la fonction P par mr si rest strictement plus grand que Ie degre de P. La fonction P f est done annulee par l'ideal sr. Soit p E S(;*). Ecrivons pe.\ sous la forme c;(u) . e.\ avec u E V::' , et considerons la fonction generalisee U· f dans U. Elle est annulee par une puissance de a d'apres ce qui precede. Elle est invariante. D'apres Ie theoreme de regularite d'Harish-Chandra rappele ci-dessus, U· fest une

180 M. Duflo and M. Very}ne

fonction localement integrable. On en deduit que u· Jet 9(peA) sont egales, puisque ces deux fonctions localement integrables sont egales dans Ungreg •

La relation (2) est maintenant evidente. •

Remarque 13 Soit jR la forme reelle de j engendree par les coracines. Soient U un ouvert completement invariant de g et 9 E <P(U). Considerons la fonction generalisee F(>., X) = 9(eA)(X) sur jR x U. Alors Fest solution dans C-OO(jR xU) du systeme d'equations differentielles considere par Hotta et Kashiwara et note NF dans [14], par. 4.6 (voir aussi [18]). Reciproquement, une telle solution provient d'une unique famille coMrente 9.

Soit J.L E j*. On note flJ l'unique element de Chol(gc)Gc tel que, pour X E j, on ait (9) jI'(X) = L e(wl',X).

wEW

Soit 9 E <P(U) une famille coMrente de fonctions generalisees sur U. Soit F l'application F : j* _ C-OO(U)G definie par F(A) = 9(eA). Les conditions (1) et (2) de la definition 7 impliquent les relations

(Coh 1) (Moyenne)

jl'F(A) = L F(wJ.L + A) wEW

pour tout A, J.L E j*,

(Coh 2) F(A) E C-OO(U)X pour tout A E j*.

Reciproquement, une telle application F provient d'une famille coherente. Plus generalement, on a la proposition suivante.

Proposition 14 Soit U un ouvert completement invariant de g. Soit P C j* un sous-groupe de j* stable par W et engendrant j* comme espace vectoriel. Soit AO E ;*. Soit F : P + AO - C-OO(U)G une application verifiant les relations

(Ooh 1) jI' F(A) = L F( WJ.L + A)

wEW

pour tout A E P + AO et tout J.L E P, (Ooh 2) pour tout A E P + AO, on a F(A) E 0-00 (U)X . Alors il existe une unique famille coherente 9 E <P(U) telle que F(A) =

9(eA) pour tout A E P + Ao.

Families coherentes 181

Demonstration. Quitte it remplacer >'0 par un element de P + >'0, on peut supposer >'0 regulier. On demontre par un argument analogue it celui du lemme 4 que toute fonction F' : P+>'o --+ C-OO(U)G verifiant (Coh 1) et (Coh 2), et telle que F'(>'o) = F(>.o), est egale it F. On considere l'unique familIe coMrente 8 E <p(U) telle que 8(eAO ) = F(>.o) et on applique ce resultat it la fonction F' definie par F'(>') = 8(eA) pour >. E P + >'0. •

La proposition 14 montre que, pour un ouvert completement invariant, la definition 7 est equivalente it la definition ci-dessous, qui est l'analogue infinitesimal de la definition 5.

Definition 15 Soit U un ouvert invariant de G. Soit >'0 E j*. Une application F : P + >'0 --+ C-OO(U)G verifiant les relations (Coh 1) et (Coh 2) sera appelee une famille coherente de fonctions generalisees definies dans U basee sur P + >'0.

On note :F(U, P + >'0) l'espace de ces familIes coMrentes.

Enon<;ons l'analogue infinitesimal du tMoreme 6. Soit U un ouvert de g. Si f E C-OO(U) est une fonction propre pour l'operateur de Casimir, on montre comme dans Ie par. 1.1 que f admet une restriction fit it U n t.

Tout element X E 9 s'ecrit de maniere unique sous la forme

(10) X = S(X) + H(X) + N(X)

ou les elements S(X) E gell, H(X) E ghyp et N(X) E gnil commutent deux a deux. Un ouvert G-invariant U de 9 est dit elliptique si, pour tout X E g,

on a S(X) E U si et seulement si X E U (voir [7J, definition 35). Un ouvert G-invariant eUiptique est completement invariant. Le theoreme ci-dessous est l'analogue du theoreme 6. II sera demontre au paragraphe 1.5.

Theoreme 16 Soit U un ouvert G-invariant elliptique de g. Soit >'0 E j*.

Soit F E :F(U, P + >'0). On suppose que pour tout>. E P + >'0 on a F(>.)lt = o. Alors F = o.

1.3 La famille principale B;rinc.

Dans ce paragraphe, nous supposons que 9 est une algebre de Lie semi­simple deployee. Soit 9 = t EI1 a EI1 n une decomposition d'Iwasawa de g.

On note M Ie centralisateur de a dans K. Le groupe M est discret, il contient r, KIM est compact et Mlr est fini. On choisit des mesures de Lebesgue dY, dA et dN sur t, a et n respectivement. On munit l'espace homogene KIM de la mesure invariante dk tangente a dY, et 9 de la mesure dX =dY dA dN.

182 M. Duflo and M. Vergne

Comme 11 est deployee, 11 est une sous-algebre de Cartan de 11, on peut choisir ; = Pc et ~ + Ie systeme de racines positives deIini par n. Le groupe de Weyl W est isomorphe au groupe NK(I1)/M, OU NK(I1) est Ie normalisateur de 11 dans K. (En fait ici, comme la famille coherente B;"inc que nous construisons dans ce paragraphe est W-invariante, Ie choix de ~ + est irrelevant).

Notons que comme 11 est deployee, 1Ihyp est l'ensemble des elements de 11 conjugues d'elements de 11. Rappelons la formule d'integration

(11) r O(X)dX = #lW r r r O(k(A + N))ID(A)I dk dA dN i8h~P iK/M io in

pour toute fonction 0 integrable sur 1Ihyp.

Soit ~ une fonction borelienne localement bornee sur 11. Notons B;"inc(~) la fonction borelienne G-invariante sur 11 qui est nulle en dehors de 1Ihypn1lreg

et telle que

(12)

pour tout A E 11 n 1Ireg. Elle est localement sommable et definit done un element de C-OO(1I)G.

Considerons l'application Mo : C~(g) ---+ C~(I1) definie pour 1/J E C~(g) par

Mo1/J(A) = r r 1/J(k . (A + N))dkdN. iK/M in

II resulte de la formule (11) que, pour toute fonction test 1/J sur 11, on a

(13) i B;"inc(~)(X)1/J(X)dX = 1 ~(A)MQ1/J(A)dA. La formule (13) permet de definir B;"inc(~) E C-OO(g)G pour toute fonction generalisee ~ E C-oo ( 11).

Si fest une fonction Coo invariante par G sur 11, sa restriction it 11 EB n verifie f(A + N) = f(A). On voit done que M. verifie:

(14)

On voit aussi facilement que Mo verifie

(15)

pour tout U E S(gc)Gc .

L'espace Chol(;) est un espace de fonctions analytiques sur 11 et il resulte de (14) et (15) la restriction de B;"inc it Chol(j) est une famille coherente.

Familles cohlrentes 183

Bien que nous n'en ayons pas besoin dans cet article, signalons que pour >. E 0*, la fonction generalisee B;"inA ei>.) est la transformee de Fourier de 1'0rbite coadjointe G.>. E g*.

La famille coherente B;"inc est caracterisee par son support. Notons ghyp l'adherence de ghyp dans g. L'ensemble ghyp est egal au sous-ensemble de 9 forme des X E 9 tels que l'on ait S(X) = o. Proposition 17 Supposons que 9 soit deployee. Soit >. un element regulier de ;*. Si f E C-OO(g)r est d support contenu dans ghyp, alors fest proportionnelle d B;"inc(e'~).

Demonstration. Soit f comme dans l'enonce. II existe des fonctions localement constantes c(w)(X) definies pour w E W et X E a n greg telles que, pour X E a n greg, on ait

(16) " ()(X) (>. w-1X) f(X) = L.JwEW ewe'

D(X) .

(Ceci est equivalent ala formule (7)). Comme tout element de West realise par un element du normalisateur de a dans G, l'invariance de la fonction f implique que pour tout w E W, tout w' E Wet X E areg on a

(17) c(ww')(X) = €(w)c(w')(w-1X).

D'autre part, on a (18) f{X) = 0

pour tout X E greg non conjugue de o.

Ces trois conditions traduisent Ie fait que f est un element de C-OO(greg)r, nul en dehors de ghyp' Le fait que f appartienne a C-OO(g)r implique des relations supplementaires: ce sont les relations de saut d'Harish-Chandra (11), lemma 18, que nous decrivons dans Ie cas particulier considere ici.

On considere l::1 comme un sous-ensemble de a*. Soit X E a n greg et soit a E l::1 une racine telle que ker a soit un mur de la chambre determinee par X. Soit Sa E W la reflexion correspondant it a. Alors, pour tout w E W, on a (19) c(Saw)(X) = c(w)(X).

On deduit de (17) et de (19) qu'il existe une constante c telle que 1'0n ait

e(>.,w- 1 X)

f{X) = c w~ ID(X)I .

•

184 M. Duflo and M. Veryne

Corollaire 18 Soit 9 E ~(g) une lamille coherente. Supposons que pour un A E ;* regulier 9(e.\) soit a support contenu dans ghyp' Alors 9 est proportionnelle a la lamille B;"inc'

Pour terminer ce paragraphe, calculons la restriction a t de la fonction generalisee B;"inc(e'\). Notons t5~ la fonction generalisee sur t telle que l'on ait (20) It5~(Y)'IjJ(Y)dY = 'IjJ(O) r dk

e JK/M

pour toute fonction test 'IjJ sur t.

Lemme 19 Pour tout A E ;*, on a

B;"inc(e.\)le = t5~. Demonstration. Soit I E C-OO(g) une fonction propre de l'operateur de Casimir. On peut calculer la restriction lie de la manj(~re suivante. Rappelons la decomposition de Cartan 9 = t EB p. On ecrit un element de 9 sous la forme Y + Z avec YEt, Z E p. Soit € > O. Soit X une fonction Coo a support compact sur p telle que Jp X(Z)dZ = 1. Soit € > 0 et soit X«Z) = cdimpX(Cl Z). Nous dirons que X<dZ est une approximation de l'identiM sur p. Si 'IjJ est une fonction test sur t on a

(21) l lle(Y)'IjJ(Y)dY = lim 1 r I(Y + Z)'IjJ(Y)x«Z)dY dZ. e <-0 e~

On choisit une approximation de l'identite x< sur p comme ci-dessus qui soit de plus K-invariante. Soit 'IjJ une fonction test sur t. Nous devons demontrer que la limite pour € tendant vers 0 de l'integrale

(22)

est egale a JK/M dkx'IjJ(O). On se ramEme immediatement au cas ou 'IjJ est K­invariante. Notons prp et pre les projections de 9 sur P et t respectivement. L'integrale (22) est egale a

1 B;"inc(e.\) (X)'IjJ(preX)X< (prpX)dX. PhilP

Appliquons les formules (11) et (12). L'integrale (22) est egale a

#1w r dk x 11 L eW.\(A)'IjJ(pr,(N»x«prp(A + N»dA dN. JK / M a RwEW

L'application prp est un isomorphisme de aEBn sur p. Elle transforme dA dN en dZ. Done x< 0 prp dA dN est une approximation de l'identite sur a + n. La limite lorsque € tend vers 0 de l'integrale ci-dessus est donc egale a JK/M dk x 'IjJ(0).

Families coherentes 185

Remarque 20 La famille F(>') = B;"inc(eA) est de restriction nulle a e\{o}. Mais sa restriction a t tout entier n'est pas nulle. Le tMoreme d'unicite 16 est donc sensiblement different du tMoreme d'unicite d'Harish­Chandra [13]. On ne suppose aucune condition de croissance sur les fonctions F(>.), mais on impose une condition d'annulation sur F(>')lt qui porte d'une part sur tous les >., et d'autre part sur t tout entier (et non seulement sur un ouvert "regulier" de t).

1.4 FamilIes s-coherentes et descente.

Pour demontrer Ie tMoreme 1, il sera utile de donner une generalisation de la notion de familles coherentes, obtenue naturellement par descente.

Soit S E Ge/l' Posons ~ = o(s), H = G(s) et He = Gc(I\:(s». Comme Gc est simplement connexe par hypothese, He est connexe. La sous-algebre ~ de 0 est reductive et de meme rang. Nous choisissons j contenue dans ~e, de sorte que 1'0n a I\:(s) E J. On pose 6.+(~e,j) = 6.(~e,j)n6.+, W~ = W~(j,j)

et D~ = n"'E<1+(~,j) Q E S(;*). Soit Ii E P. On note JJ: la fonction Hc-invariante sur ~e telle, que pour

tout X E ;, on ait (23) I:(X) = L eWJl(I\:(s»eWJl .

Dna (24)

wEW

J:(X) = IJl(I\:(s)expX)

pour tout X E ~c, OU IJl est l'element de R(Gc) de£ini par la formule (2). Soit cb : S(oc)Gc -+ S(~c)Hc I'homomorphisme de restriction. Soient U

un ouvert H-invariant de ~ et >. E j*. On note C-OO(U)f Ie sous-espace de C-oo(U)H forme des I E C-oo{U)H telles que cb{u) . 1= u{>')1 pour tout u E S{oc)Gc •

Definition 21 Soient U un ouvert H -invariant de ~ et >'0 E j*. Une famille s-coherente de lonctions generalisees definies dans U basee sur P + >'0 est une application F : P + >'0 -+ C-oo{U)H verifiant les relations

(Coh-s 1) I:F{>') = L,wEwF{WIi + >.) pour tout>. E P + >'0 et tout Ii E P,

(Coh-s 2) F{>.) E C-OO(U)f pour tout>. E P + >'0.

On note F{U, P + >'0, s) l'espace de ces familIes coherentes. Pour s = 1, on retrouve la definition 15.

Remarque 22 Dans [8] la condition (Coh-s 1) est remplacee par la condition

186 M. Duflo and M. Ve7yne

(Coh'-s 1) J:F(>..) = EWEWeWp(~(s»F(wJL + >..) pour tout>.. E P + >"0 et tout JL E P.

Cela revient au meme, car si F verifie (Coh-s 1), alors la famille F' telle que F'(>" + >"0) = e.\(~(s-l»F(>" + >"0) verifie (Coh'-s 1).

Supposons que >"0 soit regulier 1. II resulte du lemme 4 qu'une famille coherente F E :F(U, P + >"0, s) est determinee par la fonction F(>..o) E C-OO(U)Z,. Reciproquement, si U est eompletement invariant, on peut montrer comme dans la proposition 10 que pour tout element f E C-OO(U)fo il existe FE :F(U, P + >"0, s) tel que F(>..o) = f.

Supposons par exemple que s soit regulier dans G, e'est-a-dire que ~ soit une algebre de Cartan de g. On a done; = ~c. Soit >"0 E ;*. Soit v E W. Pour>" E P, posons Fv(>" + >"0) = ev.\(~(s»ev(.\+.\o}. On verifie immediatement que l'on a Fv E :F(~, P + >"0, s) et l'on obtient Ie lemme suivant.

Lemme 23 Soit s E Gell un element regulier. Les Fv (v E W) forment un base de :F(~,P + >"o,s).

Expliquons comment on obtient des familles s-eoherentes par la methode de descente. Pour tout a > 0, on note ga l'ouvert de 9 forme des X tels que, pour toute valeur propre ( de ad X on ait I im( () I < a. Plus generalement, si e est un sous-ensemble de g, on pose ta = en ga' L'ouvert ga est elliptique.

Soit s E Gell • On pose G(s)a = exp(g(s)a). L'ensemble G(s)a de G(s) est invariant par l'action adjointe de G(s) et on peut considerer l'espace fibre G XC(s) G(s)a de base G/G(s). On notera [g, h]la classe dans G XC(s} G(s)a d'un element (g, h) de G x G(s)a. Considerons l'application 'Y : G XC(s} G(s)a - G definie par 'Y([g,h]) = gshg-l. L'image de 'Y est l'ensemble (25) W(s, a) = {gshg-l, 9 E G, h E G(s)a}.

On choisit «:(s) > 0 tel que G(s).(s} soit ouvert dans G(s), l'application exponentielle un diffeomorphisme de g(S)'(8} sur G(s).(s}, W(s, «:(s» un voisinage ouvert G-invariant de la classe de eonjugaison de s dans G et 'Yest un diffeomorphisme de G XC(s} G(s).(s} sur W(s, «:(s». C'est possible, voir [7], lemme 38. On notera que ceci entraine la relation

(26)

pour tout Y E g(S)'(8}'

lSauf mention du contraire, A E j* regulier signifie regulier comme element de DC

Familles coherentes 187

On fixe un ouvert eUiptique V de G. Posons Veil = V n Gell' NollS utiliserons Ie lemme suivant ([7], lemme 40).

Lemme 24 Soit V un ouvert elliptique de G. 1) Soit s E Veil. Il existe 0 < €v(s) :S €(s) tel que l'on ait l'inclusion

W(s, €v(s)) C V. 2) Quelque so it le choix de la fonction s ~ a(s) avec 0 < a(s) :S €v(s),

la famille (W( s, a( s) ))sEGellnV est un recouvrement ouvert de V.

Si [ est une algebre de Lie reelle, on note jl la fonction analytique sur [ definie par

(27) ead(X/2) _ e-ad(X/2)

jl(X) = det I( ad(X) ).

Soient s E Gell et X E g(s). On pose

ds(X) = detg/g(s)(l- sexpX)

pour X E g(s). On a jo(s) (X) > 0 et ds(X) > 0 pour X E g(s)«s). On note

j:(s~ et d!/2 les racines carrees de jo(s) et de ds positives sur g(s)«s). Soit V un ouvert invariant elliptique de G. Soient f E C-OO(V)G, S E

Veil et 0 < a :S €v(s). Alors f admet une restriction it la sous-variete fermee sG(s)a de l'ouvert W(s, a). On notera informellement X ~ f(seX) la fonction generalisee sur g( s)a obtenue en composant la restriction et l'exponentielle. Elle determine f dans W(s, a). On note Es(J) la fonction generalisee G(s)-invariante sur g(s)a definie par:

(28)

si X E g(s)a. Employons les notations ~ = g(s), H et He du paragraphe precedent et

notons 'Y~ : Z(gc) -+ S(~c)Hc l'homomorphisme compose de l'isomorphisme d'Harish-Chandra Z(gc) f'V S(gc)Gc et de l'homomorphisme de Chevalley c~ : S(gc)Gc -+ S(~c)Hc. La definition de Es est motivee par Ie resultat suivant, dil a Harish-Chandra (corollary of lemma 24 and lemma 16 of [12]).

Proposition 25 Pour tout u E Z(gc) et toute fonction generalisee f E C-OO(V)G, on a Es(u, f) = 'Y~(u) . Es(J).

On suppose que j est une sous-algebre de Cartan de ~c. Soit AO E ;*_ Soit F E T(V, P + AO) une famille coherente de fonctions generalisees sur V basee sur P + AO- Pour A E P, on pose

188 M. Duflo and M. Veryne

Proposition 26 Soient s E Cell, 0 < a ::5: €v(s) et F E .r(V, P + AO). On a RsF E .r(,,(s)a, P + AO, s).

Demonstration. La propriete (Coh-s-l) resulte immooiatement des definitions, et la propriete (Coh-s-2) de la proposition 25. I

Notons 3 Ie centre et I I'algebre derivee de ~. Supposons que l'algebre I soit deployee. On choisit une algebre de

Cartan a de ~ telle que a n I soit deployee, et on choisit j = ac, de sorte que w~ laisse stable a. Comme dans Ie paragraphe 1.3 on construit pour tout 4> E C-OO(a) une fonction generalisee H-invariante sur ~ notee B!,.inc(4)). Si 4> est borelienne et localement bomee, c'est la fonction localement sommable H-invariante nulle en dehors des points conjugues de ~reg n 0

et qui pour A E ~reg n 0 est donnee par la formule

(29)

Soit AO E j*. Soit v E W~\w. Pour A E P posons

(30) F,,(A + AO) = eVA(K(s»B;!,.inc(eW.+>'o»,

ou vest un representant de v dans W. Ceci a bien un sens, car B;!,.inc(ev(MAo» et evA(K(s» ne dependent pas du choix du representant v. On verifie immediatement que l'on a F" E .r(~, P + AO, s).

Introduisons I'ensemble

(31) C~ = {X E ~,S(X) E 3}.

Comme I est deployee, on voit que C~ est l'adherence de l'ensemble 3 + Ihyp, ou encore l'adherence de l'ensemble des elements de ~ conjugues d'un element de o. Les fonctions generalisees B;!,.inc(¢) definies ci-dessus ont leur support contenu dans C~. Nous utiliserons la generalisation suivante de la proposition 17, dont la demonstration est analogue. Nous notons encore B;!,.inc(4)) la restriction de B;!,.inc(4)) it ~a.

Proposition 27 Soit s E Cell. On suppose que l'algebre derivee de ~ est deployee. Soit a > o. Soit A E j* un element regulier. Soit f E C-OO(~a)f. On suppose que Ie support de fest contenu dans C~ n ~a. Alors il existe des constantes d" E C telles que l'on ait f = E"Ew~\wd"B!"inc(e"A).

Le corollaire suivant contient Ie lemme 23 comme cas particulier.

Corollaire 28 Soit s E Cell. On suppose que l'algebre derivee de ~ est deployee. Soit a > o. Soit F E .r(~a, P + AO, s). On suppose qu'iZ existe un element regulier A E P + AO tel que Ie support de F(>.) soit contenu dans C~ n ~a. Alors Fest combinaison lineaire des F", avec v E W~ \W.

Families coMrentes 189

Demonstration. Choisissons un element A + AO de P + AO, regulier dans gc. D'apres la proposition 27 on a F(A + AO) = EVEW~\wdvB:"inc(etJ(~+Ao» et donc

F(A + AO) = 2: CvFv(A + AO), tJEW~\W

ou l'on a pose Cv = dtJetJ~(I\':(s»-l. On conclut en utilisant Ie fait que les familles cohCrentes sont determinees par leur valeur en un point regulier. •

Enonc;;ons un theoreme d'unicite pour les familles s-coherentes. Sa demonstration fait l'objet du paragraphe 1.5. Soit s E K. Nous employons les notations ~, H, etc ... ci-dessus. Soit U un ouvert de~. Un argument similaire a celui du paragraphe 1.1 montre que toute fonction generalisee f E C-OO (U)lftn admet une restriction flbnt a e n ~ n U. En fait, notant toujours I l'algebre derivee de ~, on voit de meme que f admet une restriction a e n I n u.

Exemple 29 Supposons I deployee. On definit de maniere analogue a la formule (20) une fonction generalisee 6bnt sur Int. Pour un certain choix dX de la mesure de Lebesgue sur I n e, on a

pour toute fonction test 1/J sur I, et

(32)

pour tout A E ;*. Soit 3 Ie centre de~. Ecrivons YEt n ~ sous la forme Y = Z + X avec

Z E en 3 et X E t n L On a de meme

(33)

Le theoreme ci-dessous sera demontre dans Ie paragraphe suivant.

Theoreme 30 Soient s E K, 0 < a :::; E( s), AO E ;* et F un element de :F(g(s)a,P + AO,S). On suppose que l'on a F(A)lg(s)nt = 0 pour tout A E P + AO. Alors on a F = O.

1.5 Demonstration des theoremes d'unicite.

Nous definissons l'analogue infinitesimal de l'application de descente RSl dans une situation un peu plus generale. Soit ~ une sous-algebre de Lie

190 M. Dufto and M. Veryne

algebrique et reductive de g. On note He Ie sous-groupe connexe de Ge d'algebre de Lie ~e et H un sous-groupe de G d'algebre de Lie ~ et tel que K,(H) soit contenu dans He. On pose ~ell = gell n~. Soit X E~. Ecrivons X = B(X) + H(X) + N(X) comme en (10). Alors B(X), H(X) et N(X) appartiennent a~. Un ouvert H-invariant U de ~ est dit elliptique si, pour tout X E ~, on a B(X) E U si et seulement X E U.

Soit B E I)ell. L'ouvert I)(B)o de I)(B) est invariant par l'action adjointe de H(B) et on peut considerer l'espace fibre HXH(S)~(B)o de base HI H(B). On notera [g, X] la classe dans H XH(s) I)(B)o d'un element (g, X) de H x I)(B)o. Considerons l'application "( : H XH(S) ~(B)o -+ I) definie par ,,([g, Xl) = g • (B + X). L'image de "( est l'ensemble

(34) W(I), B, a) = {g. (B + X), g E H, X E ~(B)o}.

Lorsque I) = g, nous ecrivons simplement W(I), B, a) = WeB, a). II existe f(B) > 0 tel que W(I), B, f(B» soit un voisinage ouvert H-inva­

riant de la classe de conjugaison de B dans ~ et "( est un diffeomorphisme de H XH(S) I)(B)€(s) sur W(I), B, feB»~. En particulier, on a H(B + X) C H(B) pour tout X E ~(B)€(s).

Les ouverts W(I), B, a) avec B E I)ell et 0 < a :::; f(B) forment une base des ouverts elliptiques de I): soit U un ouvert H-invariant elliptique de ~. Posons Uell = U n gell. Pour tout BE Uel/ on peut choisir 0 < a(B) :::; f(B) de telle sorte que W(I), B, a(B» c U et, pour tout tel choix, la famille (W(~, B, a(B»sE~.lIru est un recouvrement ouvert de U.

Soit X E ~(B). On pose

r~.s(X) = det ~/~(S) (B + X).

La fonction r~.s est strictement positive sur ~(B)«s) et on note r~;i sa racine cam~e positive sur I)(B)€(s).

Soit U un ouvert H-invariant elliptique de~. Soit B E ~ell n U, et soit 0 < a :::; a(B). Soit f E C-oo(U)H. La fonction generalisee f admet une restriction a B + ~(B)o. On notera informellement X 1-+ f(B + X) la fonction generalisee sur I)(B)o ainsi obtenue. Elle determine la restriction de f a l'ouvert W(~, B, a). On note E~.s(f) la fonction generalisee H(B)­invariante definie dans I)(B)o par

(35) E~.s(f)(X) = r!:i(X)f(B + X).

Notons c l'application de restriction de B(~c) sur B(~(B)c). Le result at suivant est d6 a Harish-Chandra [11].

Proposition 31 Pour tout u E B(~c)Hc et tout f E C-oo(U)H, on a

E~.s(u· f) = c(u) . E~.s(f).

Familles coherentes 191

Soit s E Gell • Nous employons les notations ~ = g(s), etc... du paragraphe 1.4. Soit U un ouvert H-invariant elliptique de ~ et soit S E Uell·

Onposes' = ses . C'est un element elliptique de G. On choisit 0 < a:S f(S) tel que W(~, S, a) cU.

On fait l'hypothese suivante:

(36) g(s') = g(s) n g(S).

Cette hypothese est verifiee par exemple si S E ~«s) n Uell.

On choisit j C g(s'). Soient Ao E j* et F E :F(U, P + Ao, s). Pour tout A E P + AO, on pose (37) Rs.sF(A) = E~.s(F(A)). On a Rs.sF(A) E C-OO(g(s')a).

Le lemme suivant resulte immediatement des definitions et de la proposition 31.

Lemme 32 On a Rs.sF E :F(g(s')a, P + Ao, s').

Apres ces preliminaires, nous pouvons demontrer les principaux resultats de cette partie.

Demonstration du theoreme 30. On fait la demonstration par recurrence sur la dimension de g(s). Le tMoreme est vrai si dimg(s) = 0, puisqu'il n'existe pas d'element s E K tel que dimg(s) = O.

Soit s E K. On suppose que Ie theoreme est demontre pour tous les elements s' E K tels que dimg(s') < dimg(s). Soient a, AD et F comme dans l'enonce du tMoreme. Par Ie choix de a, on a G(seX) C G(s) pour tout X E g(s)a. Soit S E g(s)a n gell. Posons s' = ses . Choisissons 0< b:S f(S') de sorte que l'on ait S+9(S')b C g(S)a. On a G(s'eS') C G(s') pour tout S' E 9(S')b. Donc on a dimg(seS') :s dimg(seS ) pour tout S' E W(g(s), S, b). On en deduit que I'ensemble des X E g(8)a tels que dimg(seS(X») < dimg(s) est ouvert dans g(s)a (S(X) est defini en (10»).

Considerons I'ensemble Cs forme des X E g(8)a tels que S(X) appartienne au centre.} de g(s). D'apres ce qui precede, il est ferme dans g(8)a.

Montrons que l'hypothese de recurrence entraine que pour tout A E P+AO, la fonction generalisee F(A) est a support dans Cs. En effet, on peut recouvrir Ie complementaire de Cs dans g(S)a par des ouverts W(g(s), S, b) tels que S E g(s)a nt, et tels que l'on ait, en posant s' = ses , dimg(s') < dimg(s). Pour un tel ouvert, considerons la famille coMrente E = Rs.sF E

:F(9(S')b, P + AD, s') (Iemme 32). Montrons qu'elle satisfait aux hypotheses du tMoreme 30. Soit f E C-OO(W(g(s), S, b») une fonction generalisee G(s)-invariante et S(g(s)c)G(sLfinie. Considerons la sous-variet8 fermee

192 M. Dufto and M. Vergne

N = tn(S+S(S')b) de W(s(s), S, b). Alors Ie front d'onde de f est transverse a N, et I'on peut definir la restriction fiN de faN. De meme, soit M = tnW(s(s),S,b). On a N c M c W(s(s),S,b), on peut definir les restrictions successives flM, (fIM)IN et I'on a (fIM)IN = fiN. On applique ceci a f = F()") avec).. E P + )..0. On en deduit que F()..)IN = 0 puisque par hypothese F()..)IM = O. Posons M' = S + 9(S')b. On applique Ie meme argument a la suite d'inclusions N c M' c W(g(s), S, b). On a donc (F{)..)IM/)IN = o. Comme on obtient Rs,sF()..) en multipliant F{)..)IMI par une fonction analytique, on voit que Rs,sF{)..)IN est nul. Par l'hypothese de recurrence, Rs,sF est nulle, et donc Fest nulle dans W{g{s), S, b).

Si Cs est d'interieur vide, comme les fonctions F{)") sont localement sommables, elles sont nulles et la demonstration est terminee.

Supposons I'ensemble Cs d'interieur non vide. Notons ~ = g{s) et r I'algebre derivee de S{s). Alors rest deployee et I'on est dans les conditions d'application du corollaire 28. Soient c" E e (v E W,,\W) les constantes telles que I'on ait F = L:vEW~\W c"Fv. II resulte de I'exemple 29 que, pour tout).. E P, la restriction de F()" + )..0) a t n r existe. Elle est egale a 0 puisque la restriction a t n ~ est nulle par hypothese. D'apres la formule (32), elle est egale a L:vEW~\Wc"eVA{I\:{s»8bnt. On a donc

(38) E c"eVA(I\:(s» = 0 vEW~\W

pour tout).. E P. II s'agit de montrer que la relation (38) entraine que les constantes c"

sont nulIes. C'est ici que nous utilisons I'hypothese que Ie groupe Gc est simplement connexe. On sait que J s'identifie au groupe des caracteres (a valeurs dans eX) du groupe P par l'application qui a j E J associe Ie caractere ).. 1-+ eA{j). On sait aussi que des caracteres distincts sont lineairement independants. II nous suffit donc de demontrer que les points v-1.I\:(s) de J sont deux a deux distincts quand v parcourt W" \W. Comme Gc est simplement connexe, Ie centralisateur de 1\:( s) est connexe. Son intersection avec West donc W,,' ce qui termine notre demonstration. •

Demonstration du theoreme 6. Soit F E .r(V, P+)..o) comme dans Ie thooreme 6. D'apres Ie lemme 24, pour demontrer que Fest nulle, il suffit de montrer que, pour tout s E K n V, la restriction de F a W{s, EV(S» est nulle. Pour cela, il suffit de montrer que, posant a = EV{S), la famille s-coherente R,.F E .r{g(s)a, P + )..0, s) est nulle (voir Ia proposition 26).

Soient s et a > 0 comme ci-dessus. Soit).. E P + )..0. On voit facilement que Ie front d'onde de Ia fonction generalisee F()") est transverse a Ia sous­variete fermee KnsG(s)a de W(s,a). La restriction de F()..) a KnsG(s)a

Familles coherentes 193

s'obtient soit comme la restriction de la restriction a K n W(s, a), soit comme la restriction de la restriction a sG(s)a' II resulte des definitions que l'on a RsF(>')lu(s)m = O. 11 resulte du thooreme 30 que l'on a RsF = O. •

Demonstration du theoreme 16. Soit F comme dans Ie theoreme 16. Pour tout 8 E gell n U tel que g(8) = g(eS), choisissons 0 < a ~ €(8) tel que W(8, a) soit contenu dans U et considerons la famille Rl,sF E

F(g(eS)a, P + >'0, eS) definie par la formule (37). Un argument semblable a celui employe dans la demonstration du theoreme 6 ci-dessus montre que la famille Rl,sF verifie les hypotheses du theoreme 30. Donc R},sF = O.

Introduisons Ie sous-ensemble A de 9 forme des X E 9 tels que l'on ait g(8(X)) = g(eS(X»). Nous laissons Ie lecteur se convaincre qu'il est ouvert dans 9 et que son complementaire est de mesure nulle.

On vient de demontrer que F(>.) est nulle dans UnA pour tout>. E

P + >'0. Comme F(>.) est une fonction localement sommable, elle est nulle dans U .•

2 Groupes non connexes.

2.1 FamilIes coherentes de fonctions generalisees: enonce du theoreme.

Dans cette section G designe toujours un groupe de Lie semi-simple reel connexe verifiant la condition expliquee dans l'introduction. Nous considerons une variete differentiable Y munie d'une action a gauche de G et d'une action a droite de G qui commutent. On suppose de plus que Y est un espace homogene principal pour l'action a droite et pour l'action a gauche.

Soit L un groupe de Lie dont la composante neutre est G. Soit x un point de L. La composante connexe Lx = xG de L contenant x fournit un exemple de tel espace. Cet exemple est universe!' En effet, soient Y comme ci-dessus et y E Y. Si 9 E G, on a

(39) yg = (J(g)y

pour un certain (J(g) E G et l'on voit immediatement que (J est un automorphisme de G. Soit A((J, G) Ie groupe produit semi-direct du groupe Z = {(JR, n E Z} avec G. On note 0 l'eiement ((J, 1) de A((J, G). L'ensemble Y(G, (J) = OG est donc une composante connexe de A((J, G), et l'application Og f-+ yg est un isomorphisme de Y (G, (J) sur Y.

194 M. Dufto and M. Vergne

Remarque 33 Cette construction montre qu'il est equivalent de se donner You bien une composante connexe du groupe des automorphismes de G. Par exemple, si G est simplement connexe, Y est Ie revetement simplement connexe de son image dans Ie groupe des automorphismes de G.

NollS noterons Ad(y), ou meme 9 1---+ ygy-t, l'automorphisme de G defini par la formule (39). Nous noterons de la meme maniere l'automorphisme correspondant de g, ge, Ge , etc... Nous noterons aussi Ad(y) Ie diffeomorphisme de Y tel que Ad(y)(gy) = yg pour tout 9 E G. On dira qu'un ouvert V de Y est G-invariant s'il est invariant par l'action adjointe y 1---+ gyg-l pour tout 9 E G. Soit V un ouvert G-invariant de Y. On note C-OO(V)G l'espace des fonctions generalisees G-invariantes dans V. On identifie ge a l'algebre des derivations sur Y invariantes a gauche (donc X E 9 opere par derivations a droite) et U(gc) a l'algebre des operateurs differentiels invariants a gauche sur Y.

Lemme 34 Soit V un ouvert G-invariant de Y, et soit y E Y. a) L'ouvert Vest stable par Ad(y). b) L'action de Ad(y) dans C-OO(V)G est l'action identique. c) Soient f E C-OO(V)G et u E Z(gc). On au· f = Ad(y)(u) . f.

Demonstration. a) Soit yg E V. Alors gy = g(yg)g-l appartient aussi a V.

b) Soit f E C-OO(V)G. L'argument donne en a) suffit a montrer que f est invariante par Ad(y) si fest une fonction, mais pas si fest une fonction generalisee. Soit donc f E C-OO(V)G. Pour toute densite test a sur Y et tout 9 E G on a fy f(yg)da(y) = fy f(gy)da(y). Pour toute densite test f3 sur G on a donc

fa I f(yg)da(y)df3(g) = fa I f(gy)da(y)df3(g).

Soit y E Y et faisons tendre avers la mesure de Dirac en y. Comme les fonctions fG f(yg)df3(g) et fG f(gy)df3(g) sont continues (et meme differentiables) par rapport a y, on a Ie droit de passer a Ia limite et 1'0n obtient Ia relation fG f(yg)df3(g) = fG f(gy)df3(g). Ceci signifie que fest invariante par Ad(y).

c) Ceci resuite de b). •

Remarque 35 Un maniere equivalente d'exprimer Ie point c) du Iemme 34 est de dire que I'action a droite et l'action a gauche de Z(gc) dans C-OO(V)G cOincident, ce qui repond a une question bien naturelle.

Families cohirentes 195

Notre but est de generaliser aux fonctions generalisees f E C-OO(V)G les resultats de la premiere partie. Pour cela nous avons besoin de quelques preliminaires sur les fonctions rationnelles invariantes sur Y.

On pose Ye = Y XG Ge, ou 9 E G agit sur Ge par la multiplication a gauche par ~(g). C'est un espace homogene principal pour l'action a droite de Ge . On note encore ~ l'application canonique de Y dans Ye. Pour y E Y on note encore Ad(y) l'automorphisme de Ge dont la differentielle est l'extension de Ad(y) a Dc. II existe une application (notee encore Ad(.)) de Ye dans Ie groupe des automorphismes de Ge telle que Ad(yg) = Ad(y) Ad(g) pour tout y E Y et tout 9 E Ge. On definit une action a gauche de Ge dans Ye par la formule gy = yAd(y-l)(g). Cette action est holomorphe, elle commute a l'action a droite et ~ verifie la relation ~(gyg') = ~(g)~(y)~(g') pour tout 9 E G, y E Y et g' E G.

NOllS noterons F(Yc) l'algebre des fonctions </> sur Ye telles que, pour un element y EYe, la fonction 9 t-+ </>(yg) soit dans F(Gc). Cette condition ne depend pas du choix de y et F(Yc) est stable par l'action a droite et a gauche de Ge. NOllS noterons F(yc)Gc la sous-algebre des elements invariants par l'action adjointe.

Rappelons que tout automorphisme de De laisse stable une sous-algebre de Borel de De (Steinberg [19]). Soit y EYe. Choisissons une sous-algebre de Borel b de De stable par Ad(y). Soit a une sous-algebre de Cartan de b. Identifiant a au quotient de b par son radical unipotent, on obtient une action de Ad(y) dans a et dans son dual a* qui laisse stable Ie systeme des racines A(De, a) et l'ensemble de racines positives defini par b.

Rappelons que nous avons choisi une algebre de Cartan j et un systeme de racines positives A+ C A = A(De,j). En transportant Ad(y) par l'unique element de W(j, a) qui conserve les systemes de racines positives, on obtient un automorphisme de j que nollS noterons (j. II ne depend pas des choix faits de y E Y et b stable par Ad(y). Nous noterons aussi (j

l'automorphisme correspondant de J, j*, etc ... II induit un automorphisme du systeme de racines A qui conserve A +, et un automorphisme de P.

Si y E Ye est semi-simple (c'est-a-dire si Ad(y) E L(Dc, Dc) est semi­simple), on peut choisir l'algebre de Cartan a introduite ci-dessus stable par Ad(y) (voir [19]). Introduisons Ie sous-ensemble YJ de Ye forme des elements y E Yc qui normalisent j et tels que l'action induite sur A normalise A +. II resulte de ce qui precede que YJ est un espace homogene principal pour l'action a droite et a gauche de J sur YJ , et que pour tout y E YJ l'action de Ad(y) dans jest egale a (j. Les elements semi-simples de Yc sont ceux dont la classe de conjugaison (sous l'action adjointe de Gc) rencontre YJ •

Rappelons Ie lemme suivant (dont la demonstration est simple). On

196 M. Duflo and M. Veryne

note Jq l'ensemble des elements de J de la forme (1-1(j)r1 avec j E J. Quel que soit y E YJ, c'est l'ensemble des elements y-ljyj-l ou j parcourt J. On note ;q Ie sous-espace «(1-1 - 1)(;) de;. Comme (1 est d'ordre fini, il est semi-simple, et ;q est l'orthogonal dans; de l'espace ;«(1) des points fixes de (1.

Lemme 36 a) L'ensemble Jq est un sous-groupe ferme de J, stable par (1. Son algebre de Lie est ;u. L'algebre de Lie de J/Jq est canoniquement isomorphe a ;«(1).

b) Le groupe des camcteres de J / Jq s'identi/ie au sous-groupe P( (1) des points fixes de (1 dans P.

c) Les classes de conjugaison de J dans YJ sont les ensembles yJq = Jqy.

Nous noterons YJ / Jq l'espace des classes suivant Jq • 11 est egal a l'espace des orbites de l'action adjointe de J. L'action a gauche et a droite de J dans YJ / Jq sont egales, et font de YJ / Jq un espace homog€me principal de groupe J / Jq •

Nous noterons F(YJ) l'algebre des fonctions obtenues a partir de F( J) par la translation par un element y E YJ et F(YJ V = F(YJ / Ju ) la sous-algebre des elements J-invariants. Chaque point y E YJ definit un isomorphisme de F(YJ)J sur F(J/Jq ). Plus explicitement, pour tout >. E P«(1) on note ey,oX l'element de F(YJ) tel que ey,oX(yj) = eoX(j) pour tout j E J. Alors les ey,oX (avec>. E P«(1» forment une base de F(YJV.

Soit y E YJ • On a ;«(1) = ; n ge(Y). Comme,6,+ est stable par (1, ;«(1) contient des elements reguliers de g. Donc; est Ie centralisateur de ;«(1) dans g, et ;«(1) est une sous-algebre de Cartan de l'algebre de Lie reductive gc(y). Par definition, un element de Ye est dit regulier s'il est semi-simple et si dimgc(y) est minimum. Notons YCreg l'ensemble des elements reguliers de Yc. Alors YCreg n YJ est egal a l'ensemble des elements y E Yc tels que gc(y) = ;«(1) (voir [19]).

Soient NJ = N(Ge, J) Ie normalisateur de J dans Ge et NYJ = N(Gc , YJ ) Ie normalisateur de YJ dans Ge. Donc on a W = NJ/J. Le lemme suivant est facile.

Lemme 37 On a J C NYJ C N J et N yJ / Jest egal au sous-groupe W«(1) des points fixes de (1 dans W.

Le groupe W«(1) = NyJ/J opere donc dans l'espace YJ/Jq des orbites adjointes de J dans YJ •

Soit y E YJ • Rappelons Ie resultat fondamental suivant qui vient de ce que Ge est simplement connexe et y semi-simple (voir [19]).

Familles coherentes 197

Theoreme 38 Soit y E YJ. Le groupe Gc(y) est connexe.

Notons WI! Ie groupe de Weyl de gc(y). Comme Gc(y) est connexe, on a WII = N(Gc(y),;(u))fJu. Comme un element n E Gc qui normalise ;(u) normalise aussi ; et J, Ie groupe WI! est un sous-groupe de W. Comme d'autre part N(Gc(Y),;(u)) normalise YJ, WII est en fait un sous-groupe de W(u).

Lemme 39 Soit y E YJ et soit y = yJu l'orbite adjointe de y so'us l'action de J.

a) Le stabilisateur de y dans W(u) est egal a WI/' b) Soit 011 C Yc l'orbite adjointe de y sous l'action de Gc. L'ensemble

011 n YJ est egal a l'orbite de y sous l'action de NYr c) L 'application n I-t nyn-1 de NYJ dans YJ induit une bijection de

W(u)/WII sur l'espace (011 n YJ)/Ju'

Demonstration. Le a) resulte immediatement de ce qui precooe. Demontrons b). II s'agit de prouver que deux points y et y' de YJ conjugues par un element 9 E Gc sont conjugues par un element du normalisateur de YJ. On a gyg-l = y', et donc Ad(g);(u) est une sous-algebre de Cartan de gc(y'). Soit h E Gc(y') tel que Ad(h) Ad(g);(u) = ;(u). Alors n = hg est un element de NYJ tel que nyn-1 = y'. Le c) resulte de a) et b). •

Dans Ie meme ordre d'idees, nous nous servirons du lemme suivant.

Lemme 40 a) Soit W>. c j* une orbite de W stable par u. Alors W>. n ;*(u) est non vide.

b) Soit >. E ;*(u). On a W>. n ;*(u) = W(u)>..

Demonstration. a) Soit hOt E ; la coracine associee it une racine a E ~. Nous dirons qu'un element>. E ;* est dominant si, pour tout a E ~ +, on a ou bien Re >'(ho:) > 0 ou bien Re >'(ho:) = 0 et 1m >'(ho:) ;::: O. Soit >. E ;* un element dont l'orbite sous West stable par u, et soit >.+ l'unique element dominant de W>.. On a u(>.+) = >.+.

b) En rempla«;ant >. par >.+ comme ci-dessus, on voit que l'on peut supposer de plus >. dominant. Soit>.' E w>.n;*(u). L'ensemble des wE W tels que w>.' = >. est stable par u et consiste en une seule classe W(>.)w a gauche modulo Ie stabilisateur de >.. II existe dans W(>.)w un unique element w' de longueur minimale. L'element w' est stable par u puisque ~ + est stable par u. •

Le groupe W(u) opere dans F(YJ)J. Si ¢ est un element de F(YJ)J et si w E W(u), nous notons W· ¢ Ie transforme. Par exemple, soient y E YJ

et >. E P(u). Posons y = yJu • La fonction ell.>' ne depend que de y. On la

198 M. Duflo and M. Vergne

notera aussi eii,A' On a w . eii,A = eWii,WA' Si y et z sont deux elements de YJ/JlT on note z/y 1'eiement t de J/JlT tel que z = yt. On a

(40)

En particulier, avec les notations ci-dessus, on a

(41) w . eii,A = eA(y/wy)eii,WA

Soient y E YJ!JlT et JL E P(a). On pose

iii,,. = E ewii,wp' WEW(lT)

Il resulte de ce qui precooe que pour y fixe et J.L parcourant P+ n P(a), ces fonctions forment une base de 1'espace F(YJ )NYJ = F(YJ / JlT ) W(lT).

Donnons une description de 1'espace F(yc)Gc en terme de representa­tions. Soit y EYe. On notera encore a la classe de 1'automorphisme Ad(y) (de Ge ou Dc suivant Ie contexte) modulo Ie groupe des automorphismes interieurs Ad(Gc). Pour une representation rationnelle r de dimension finie de Ge dans un espace vectoriel complexe V, la representation 9 -+ r(y-lgy) est notee Yr. La classe d'equivalence de Yr ne depend pas de y E Y, mais seulement de a. On la notera lTr. Soit V l'espace de r. L'espace de Yr est encore V. Cependant nous Ie noterons parfois YV pour indiquer qu'il est muni de la representation Yr. On note LGc(YV, V) 1'espace des endomorphismes a de V tels que ar(y-lgy» = r(g)a pour tout 9 E Ge.

On note Ayc(V) 1'espace des applications A de Yc dans L(V, V) telles que 1'on ait A(gyg') = r(g)A(y)r(g') pour tout y EYe, 9 E Ge et g' E Ge. Le lemme suivant est laisse au lecteur.

Lemme 41 Soit y E Yc. Si A est un element de Ayc(V), on a A(y) E LGc(YV, V), et l'application A ....... A(y) est une bijection de Ayc(V) sur LGc(YV, V). En particulier Ayc(V) est non nul si et seulement si les representations IT r et r ne sont pas disjointes.

A chaque A E Ayc(V) on associe un element Tr A E F(yc)Gc par la formule (Tr A)(z) = Tr(A(z» pour z EYe.

Supposons V irreductible. Alors l'espace Ayc(V) est de dimension 1 si r et IT r sont equivalentes, et il est nul sinon. Considerons en particulier la representation irreductible V (..\) de poids dominant ..\ E P+. Les representations lTV(,,\) et V(a..\) sont equivalentes, et donc lTV(,,\) est equivalente a V("\) si et seulement si ..\ E P(a) n P+. Pour tout ..\ E P(a) n p+ 1'espace Ayc(V(..\» est de dimension 1. Soit y E YJ . Nous choisissons un element non nul particulier Ay(..\) E Ayc(V(..\» en imposant la condition Ay(..\)(z)v = ey,A(z)V si v E V("\) est de poids ..\ et z E YJ .

Families coherentes 199

Lemme 42 80it y E YJ. Les fonctions Tr Ay(>..) (>.. E P(a) n P+) forment une base de l'espace F(yc)Gc.

Demonstration. Soit y EYe. On peut ecrire tout element de F(Yc) comme une combinaison lineaire de fonctions yg -+ Tr(aT>.(g)), avec a E L(V(>"), V(>")). Cette fonction est invariante si et seulement si a E LGc(YV(>"), V(>"». •

Proposition 43 L'application de restriction est un isomorphisme de F{yc)Gc sur F{yJ)NYJ = F(YJ/J<T)W(<T).

Demonstration. L'argument est standard. L'injectivite vient de ce que l'ensemble des elements semi-simples de Ye est dense et que toutes les orbites semi-simples rencontrent YJ . Pour demontrer la surjectivite, on utilise Ie lemme 42. Soit y E YJ et soit y = yJ<T' On remarque que pour tout j..t E p+ n Pea), la restriction de Tr Ay{j..t) it YJ est une combinaison lineaire Lv c,Ju,v on v parcourt l'ensemble P{V(j..t» n P+ n Pea) des poids de V{j..t) qui sont it la fois dominant et a-invariants. De plus cJJ est non nul. Considerons la distance sur P provenant de la forme de Killing de De. En raisonnant par recurrence sur la longueur de j..t, on voit que iii,JJ est combinaison lineaire de restrictions de fonctions Tr Ay{v) avec v E P{V{j..t» n P+ n Pea). •

Nous noterons fii,JJ l'unique element de F{Yc)Gc dont la restriction it YJ est iii,W (Comparer avec la formule (2)). On a

(42) hI' = fWii,wJJ

et (43)

Soit V{YJ / J<T) l'algebre des operateurs differentiels de F{YJ / J<T)' L'algebre de Lie ;(0') de JjJ<T opere par derivations dans F(YJ/J<T), S(;{a» s'identifie it l'algebre des operateurs differentiels de F(YJ / J<T) invariants par J/J<T et V(YJ/J<T) est engendree par F(YJ/J<T) et S{;(a)). Pour chaque >"0 E ;*(0'), il existe (it isomorphisme pres) un et un seul module simple M>'o

pour V{YJ/J<T) contenant un vecteur v f:. 0 tel que u·v = u{>"o)v pour tout u E S(;{a». On peut Ie decrire ainsi. Fixons un point y E YJ. Alors M>'o

admet une base ey,>. on >.. parcourt Pea) + >"0, et on a

(44)

pour tout u E 8(;(0')), tout>.. E P(a) + >"0 et pour tout j..t E Pea). Le module M>'o ne depend que de P(a) + >"0.

200 M. Duflo and M. Veryne

Soit V(Yc) l'algebre des operateurs differentiels de F(Yc) et soit 'DYe la sous-algebre engendree par F(yc)Ge et Z(gc). Notons h;" l'homomorphisme de Z(9c) dans S(;(O"))W(/7) obtenu en composant l'homomorphisme d'Harish-Chandra de Z(gc) dans S(;)W avec la projection O"-invariante de S(;)W dans S(;(O"))W(/7). Il existe un unique homomorphisme d'algebres encore note h;" de 'DYe dans V(YJ / J(7) qui cOIncide avec la restriction de Yc a YJ sur F(yc)Ge et dont la restriction a Z(gc) est l'application h;". Ceci resulte des resultats sur la composante radiale des operateurs differentiels Gc-invariants sur Yc qui se trouvent dans Bouaziz [5).

NollS utiliserons la generalisation suivante du lemme 4. La demonstration en est identique.

Lemme 44 Soit Ao un element regulier de ;(0")*. hj"(VYe)e,,,>.o·

On a M>'o =

Soit V un ouvert G-invariant de Y. Si A E j*, on note C-OO(V)X Ie sous­espace de C-OO(V)G forme des fonctions f verifiant u . f = x>.(u)f pour tout u E Z(gc). Il resulte des lemmes 34 et 40 que C-OO(V)X = 0 si A n'est pas conjugue d'un element de ;*(0"), et que les caracteres infinitesimaux interessants sont parametres par les orbites de W(O") dans ;*(0").

La composition avec K est un isomorphisme de F(Yc) sur une sous­algebre de fonctions analytiques sur Y. Par multiplication, l'espace C-OO(V)G est un module sur F(yc)Ge. D'autre part, la restriction a YJ donne un morphisme injectif de l'algebre F(yc)Ge dans F(YJ)J. On peut donc considerer tout F(YJ)J-module comme un F(Yc)Ge-module. Soit AO E ;*(0"). Le F(YJV-module M>.o introduit ci-dessllS est donc aussi un module sur F(yc)Ge. Nous disposons de tous les ingrooients necessaires pour generaliser la definition 3.

Definition 45 Soit Ao E ;*(0"). Soit V un ouvert G-invariant de Y. On appelle famille coherente de fonctions genemlisees sur V basee sur P ( 0" ) + Ao une application \}I : M>'o _ C-OO(V)G verifiant les conditions suivantes.

(1) L'application \}I est un morphisme de F(yc)Ge-modules. (2) Soit y E YJ. Pour tout A E P(O") + Ao on a \}I(e",>.) E C-OO(V)X.

On note ~(V, P(o")+Ao) l'espace de ces familIes coMrentes. Nous donnerons un exemple de famille coMrente a la fin de ce paragraphe, et un autre au paragraphe 2.2.

Soit AO un element regulier de ;(0")*. II resulte du lemme 44 qu'une familIe coMrente \}I E ~(V, P(O") + AO) est determinee par la fonction generalisee \}I(e",>.o) E C-OO(V)Xo. Reciproquement, si Vest completement

Familles coherentes 201

invariant, on peut demontrer que pour tout element f E C-OO(V)Xo il existe une famille coMrente q, E «P(V, P(O') + Ao) telle que l'on ait q,(ey,AO) = f.

Nous dirons que y E Y est semi-simple si Ad(y) est un element semi­simple de L(g, g), et que y est elliptique si Ad(y) est contenu dans un sous-groupe compact de GL(g). Nous noterons Yell l'ensemble des elements elliptiques de Y. Tout element y E Y s'ecrit de maniere unique sous la forme (45) y = s(y)eH(Y)eN(Y)

avec s(y) E Yell, H(y) E 9hyp et N(y) E gnil commutant deux it deux. Ceci resulte facilement de ce que Ad(Y) est une composante connexe du groupe des automorphismes de g. Un ouvert G-invariant V de Y est dit elliptique si pour tout y E Y, on ayE V si et seulement si on a s(y) E V.

On note YK Ie sous-ensemble de Y forme des y E Y tels que Ad(y) normalise K. Comme K est son propre normalisateur dans G, Y K est un espace homogene principal de K sous l'action it droite, et aussi sous l'action it gauche. Un element y E Y est elliptique si et seulement s'il est conjugue par G d'un element de YK. Une fonction generalisee f definie dans un ouvert V de Y et valeur propre de l'operateur de Casimir admet une restriction f!YK it YK n V.

Nous pouvons enoncer Ie tMoreme principal de cette partie. Ii sera demontre dans Ie paragraphe 2.4. Lorsque Y = G, il se reduit au tMoreme 6.

Theoreme 46 Soit V un ouvert elliptique G-invariant de Y. Soit Ao E ;*(0'). Soit q, E «P(V, P(O') + Ao). On suppose que pour tout </> E MAO on a q,(</»!YK = O. Alors q, est nulle.

Exemple 41 Considerons Ie groupe G = SL(2, JR) et soit Y = SL -(2, JR) l'ensemble des matrices de determinant -1. Tout element de Y est conjugue

it exactement un element de la forme s(t) = (~ _~-t), avec t > O. Les

elements elliptiques de Y forment une seule orbite: ils sont tous conjugues

de la matrice s = (~ ~ 1 ).

L'automorphisme Ad( s) de sl(2, C) est interieur et on a 0' = 1. On peut dire que "Y est dans la classe d'Harish-Chandra". Cependant, Ad(s) induit un automorphisme de K = SO(2, JR) qui est exterieur, et la situation n'est pas exactement celle de la premiere partie.

On choisit pour J Ie groupe des matrices diagonales de determinant 1

de sorte que YJ est l'ensemble des matrices de la forme Sz = (~ _~-l)'

202 M. Duflo and M. Veryne

avec z E Cx •

L'espace F(YJ ) est engendre par les fonctions zn, avec n E z. L'espace des fonctions obtenues en restreignant les elements de F(Yc)Gc est engendre par les fonctions zn + (_I)nz-n, avec n E N, et aussi par les fonctions

. zn+l + (_I)nz-(n+l) Flln(n) = .

z+ Z-l

Les fonctions ±F1in(n) sont les restrictions a Yc des caracteres des deux representations de dimension n + 1 du groupe SL ±(2, C).

Remarquons que les seuls ouverts elliptiques invariants de Y sont Y et I'ensemble vide. Cherchons les familles coherentes dans Y. Fixons Lo E

C. Une famille coherente "basee sur Z + Lo" sur Y est equivalente a la donnee d'une famille F(L) de fonctions analytiques et G-invariantes sur Y, parametree par 1 E Z + 10 • La condition 2) dit (pour 1 :I 0) qu'il existe des constantes a(l) et b(l) telles que l'on ait F(l)(s(t» = £~~:.l. avec f(l)(t) = a(l)elt + b(l)e-lt • La condition 1) s'ecrit:

(a(l + n)e(l+n)t + b(L + n)e-(l+n)t) + (-It (a(l - n)e(l-n)t + b(l - n)e-(l-n)t)

= (a(l)elt + b(l)e-1t)(ent + (-Ite-nt )

pour tout 1 E Z + 10 et tout n E Z. On en deduit qu'il existe des constantes a et b telles que I'on ait a(n + lo) = a et b(n + lo) = (-I)nb pour tout n. Finalement, on voit que I'on obtient une base {F+, F-} de ces familles coherentes en posant (avec les notations evidentes) f+(l)(s(t)) = e1t et f-(l)(s(t)) = (_I)I-IOe-lt • La restriction de la fonction F(l) = aF+(l) + bF-(l) a I'orbite des elements elliptiques de Y est la constante a+( _I)l-lob. Si cette restriction est nulle pour tout 1 E Z + lo, on a a + (-l)nb = 0 pour tout nEZ, et donc a et b sont nuls, ce qui illustre Ie theoreme 46.

Cet exemple montre pourquoi nous avons impose dans cet article I'hypothese que Gc est simplement connexe. Supposons un moment que nous remplacions l'homomorphisme /'i, : SL(2, JR) ---+ SL(2, C) de l'introduction par l'homomorphisme /'i, : SL(2, JR) ---+ PSL(2, C) = SL(2, C)/ ± 1. Les familles coherentes seraient alors indexees par 2z + lo, et Ie thooreme 46 serait faux pour ces nouvelles familles coherentes sur Y. Par exemple, considerons la famille correspondant aux caracteres des representations de dimension finie paire de SL(2, C), c'est-a-dire celles qui ne proviennent pas de PSL(2, C). Elle est stable par multiplication par les caracteres des representations de PSL(2, C). Elle est coherente au sens nouveau, et sa valeur en s est identiquemel1t nulle.

II est naturel de considerer aussi I'homomorphisme /'i, : PSL(2, JR) ---+

PSL(2, C) et d'etudier dans ce cadre les familles coherentes sur PSL - (2, JR).

Familles cohirentes 203

Il se trouve que Ie thooreme 46 redevient valable. Nos thCoremes d'unicite ne sont donc sans doute pas les meilleurs possibles, et les hypotheses faites tiennent peut-etre plus a la technique de demonstration qu'a la nature du sujet. Cependant, ils suffisent aux applications que nous avons en vue.

2.2 Un exemple de famille coherente : la formule de Kostant.

Dans ce paragraphe, nous donnons l'exemple de la famille coherente correspondant aux caracteres des representations de dimension finie de G qui sont u-stables, et definissons l'analogue des familles cohCrentes de modules. Ceci illustre nos definitions, mais n'est pas utilise dans la demonstration du thooreme 46.

Soit {Ie = n- EB j EB n+ la decomposition triangulaire de {Ie correspondant au choix de b,+. Si y E YJ, on pose

(46)

et (47) 8(y) = detn+(Ady).

Soit w E W(u). On pose

(48)

Les fonctions dn-, 8 et ew sont des elements de F(YJ)J. immediatement

de sorte que l'on a demontre Ie lemme suivant.

Lemme 48 La fonction 8~_ est W(u)-invariante.

Voici d'autres lemmes du meme geure.

On a

Demonstration. On ecrit n- = (n- n w(n-)) EB (n- n w(n+)) et w(n-) = (n- nw(n-)) EB (n+ nw(n-)). On verifie que l'on a detn-nw(n+)(l- Ady) = detn+nw(n-)(l- (Ady)-l) = detn+nw(n-)(Ady - 1) detn+nw(n-)((Ady)-l). Comme la dimension de n+ n w(n-) est egale a la longueur de w dans Ie groupe W, Ie lemme en resulte. •

204 M. Duflo and M. Vergne

Lemme 50 Soit w E W(a). Soit'Ii E YJ/Ju. II existe l(w) E {±1} tel que I'on ait {w = l(w)ewfj,wp/efj,p. En d'autres termes, si y E YJ represente 'Ii et si n E NYJ represente w, on a

Demonstration. En raisonnant comme dans la demonstration du lemme 49, ou bien en utilisant les lemmes 48 et 49, on voit que l'on a {! = w . 8/8. La fonction ~,p est proportionnelle it 8, et done les fonctions rationnelles ew et ewfj,wp/ ey,p ont meme carre. •

NollS poserons (49) €u(w) = €(w)€'(w)

pour tout w E W(a). On a donc

(50)

En particulier, €u est un caractere du groupe W(a) . Nous Ie calculons plus bas (proposition 55).

Theoreme 51 Soit y E YJ et soit 'Ii = yJu. 1) Soit 4> E F(YJ/Ju). II existe un unique element wtin (4)) E F(yc)Gc

tel que, pour tout element Z E YJ n YCreg on ait

w/in (4))(Z) = EWEW(U) €u(w)4>(w- 1(z» II ey,p(z)dn- (z)

_ EWEW(U) €(w)4>(w- 1(z»ewfj,wp(z)-1{w(z) - dn-(z)

2) En composant la famille wtin avec I'application '" : Y -t Yc , on obtient une famille coherente de fonctions analytiques sur Y, basee sur P(a).

3) Soit'\ E P+ n P(a) et supposons ,\ regulier, de sorte que ,\ - pest Ie plus haut poids d'une representation. On a

W£in(efj,A) = Tr Ay('\ - p).

4) Soit ,\ E P(a), et supposons ,\ non regulier. On a wtin(ey,A) = o. 5)Soient 4> E F(YJ/Ju) et wE W(a) . On a

w£in(w-1 .4» = €u(w)w£in(4».

En particulier, soient'\ E P(a) et wE W(a). On a

,T./in( ) _ ()'T./in( ) '*' II ey,A - €u W '*' y ewY,wA·

Families coherentes 205

Remarque 52 Soient A E p+ n P( a) et z E YJ n Yereg. On a done la formule

Tr AII(A)(Z) = EWEW(U) €u(w)ewii.w(.\+p)(z) = EWEW(u) €(W)eWii.WA(Z)(w(z). eii.p(z)dn-(z) dn-(z)

(51) Cette formule est equivalente a la formule pour Ie caractere des representations irreductibles des groupes compacts non connexes donnee par Kostant dans [15]. La demonstration que nous faisons ci-dessous est, comme celle de Kostant, basee sur Ie tMoreme de Bott [4].

Demonstration. Soit 7r une representation rationnelle du groupe J dans un espace de dimension finie W. Notons AYJ (W) l'espace des applications a de YJ dans L(W, W) qui verifient a(jyj') = 7r(j)a(y)7r(j') pour tout j E J, j' E J et y E YJ • Soit a E AyJ(W). Soit B- Ie sous-groupe de Borel de Ge dont l'algebre de Lie est b = n- E9;. On etend 7r en une representation rationnelle de B- triviale sur Ie radical unipotent. Soit YB - Ie normalisateur de B- dans Ye . II est egal it B-YJ = YJB-. On prolonge a en une fonction sur YB - it valeurs dans L(W, W) en posant a(bzb') = 7r(b)a(z)7r(b') pour b E B-, b' E B- et Z E YJ -il est facile de verifier que cette definition a un sens-.

On considere l'action a gauche de B- dans Wx Ge donnee par b(w,g) =

(7r(b)w, bg) et l'espace des orbites E = B-\(W x Gc). C'est un fibre equivariant de base B-\Ge pour l'action it droite de Ge . Les sections globales s'identifient aux fonctions holomorphes f : Ge f-t W verifiant f(bg) = 7r(b)f(g) pour b E B- et 9 E Ge. Soit Z E YB -. On considere l'adion de Z dans E definie par z(w,g) = (a(z)w,zgz-l). Elle induit sur l'espace des sections Paction A(z) definie par la formule (A(z)J)(g) = a(z)f(z-lgz). Pour 9 E Ge notons T(g) Paction (T(g)J)(h) = f(hg). On verifie que A s'etend en une action de Ye de telle sorte que 1'0n ait A(gzg') = T(g)A(z)T(g') pour z E YB -, 9 E Ge et g' E Ge. De meme, on definit des operateurs A.(z) et 1i(g) dans les espaces de cohomologie Hi = Hi(B-\Ge , £) ou £ est Ie faisceau des germes de sections holomorphes de E.

Rappelons que les espaces Hi sont de dimension finie, et sont nuls si i > dim(B-\Gc). On definit un element w'(a) E F(yc)Gc en posant

w'(a)(z) = L) _l)i Tr Ai(Z). i

Si z E YJ est regulier, les points fixes de l'action de Ad(z) dans B-\Gc sont les points Pw = B-\B-n, ou n parcourt un systeme de represent ants de W(a) dans NJ , et on peut calculer W'(a)(z) par la methode des points

206 M. Duflo and M. Vergne

fixes. Posons t/J{z) = Tra{z). La trace de l'action de z dans la fibre Ew de E au dessus de pw est egale a t/J{nzn-1). L'espace tangent en B- a B-\Gc s'identifie a n+, et on a det n+{l- Ad{z)-l) = dn-{z). Le determinant de l'action de 1 - Ad{z)-l dans l'espace tangent complexe a B-\Gc au point pw a dn-(nzn-1). La formule "de Lefschetz" d'Atiyah-Bott [1, 2] donne la formule

w'(a)(z) = E t/J(w(z» = LWEW(lT) E(W)t/J(W(Z»~w-l(Z) WEW(lT) dn-{w{z» dn-{z)

Comme W'(a) ne depend que de t/J, nous posons w'(t/J) = W'(a). Posons ¢ = ey,pt/J. On a donc

(52)

Comme toute fonction ¢ E F(YJ)J est de la forme ¢(z) = ey,p{z) Tra{z) pour un choix convenable de 7r et de a, ceci demontre l'assertion 1) du theoreme.

Soient V l'espace d'une representation de dimension finie rationnelle de Gc et Av E Ayc{V). On considere la restriction de cette representation a B-, ainsi que la restriction av de Av a YJ. Les fibres B- \ ({V ® W) x Gd et V ® (B-\{W x Gd) sont isomorphes par un isomorphisme qui conserve les actions de z E Yc deduites de Av et a. On en deduit facilement la condition de coherence 1) de la definition 45 pour la famille wtin. Soit >. E P{a). On considere maintenant Ie cas ou West Ie module ICA de dimension 1 dans lequel J opere par Ie caractere eA. On choisit a = t/J = ey,A. Supposons >. dominant. Le theoreme de Bott dit que Hi est nul pour i > 0, et HO est, comme Gc-module, isomorphe a V{>.). L'evaluation en 1 donne une element 8 du dual V{>.)* de V{>.). II est de plus bas poids ->., et c'est un point fixe de l'action du transpose de l'operateur Ao{Y). On en deduit que Ao, qui est proportionnel a Ay{>'), lui est egal. Ceci prouve l'assertion 3) du theoreme.

Supposons >. + p singulier. Le theoreme de Bott [4] dit que Hi est nul pour tout i, et donc wtin(ey,A+p) = 0, ce qui demontre l'assertion 4).

L'assertion 5) est evidente. II reste a demontrer la seconde condition de coherence, qui peut

s'enoncer en disant que wtin(ey,A) a Ie caractere infinitesimal defini par >.. Cela resulte par exemple des assertions 3), 4) et 5). •

Remarque 53 Le theoreme 51 montre qu'il petit etre avantageux de definir les familles coherentes en remplac:;ant la condition (2) de la definition 45 par la condition

Families coherentes 207

(2') Pour tout ,\ E P(a) +'\0 et tout ¢ E MAo on a w(¢) E C-OO(V)f+p •

Notons «I>'(V, Pea) + '\0) l'espace de ces familles coherentes. On definit une action de W( 0') par la formule w 0 ¢ = ew w . ¢, et une action dans «I>'(V, Pea) + '\0) en posant w 0 w(¢) = w(w-1 0 ¢).

On peut alors definir une famille coherente 'lit/in E «I>'(V, Pea)) sans avoir a choisir un point de base y par la formule

\fl'lin(¢) = LWEW(u) f(W)W 0 ¢. dn-

La construction faite dans la demonstration du theoreme 51 en donne une interpretation geometrique et en termes de representations. La famille 'lit/in verifie W 0 'lit/in = f(W)Wt/in .

Cependant, dans la suite de cet article il est un peu plus simple de travailler avec la definition 45.

Corollaire 54 Soient'\ E P(a), y E YJ/Ju et z E YJ. Si'\ est singulier, ou si zest singulier, on a

L fu(w)ewli.WA(Z) = o. wEW(u)

Demonstration. Si ,\ est singulier, cela resulte de l'assertion 4) du theoreme 46, et si zest singulier, de l' assertion 1), jointe a l' egalite dn- (z) = o .•

Proposition 55 On a fu( w) = det j(U) (w) pour tout w E W( 0').

Demonstration. Pour toute racine simple 0: E ~ on choisit des elements Xa et X-a de {lc de poids respectifs 0: et -0:, et tels que o:([Xa, X-a]) = 2. L'automorphisme 0' permute les racines simples, et il existe un unique au­tomorphisme ( de {Ie tel que ((X±a) = X±u(a) pour toute racine simple. Choisissons un point y E YJ tel que Ad(y) = (. II resulte de [19] que tout element w E W(a) peut titre represente par un element n E Gc(y). Donc W(a) est egal au groupe de Weyl Wy de Dc(y). Soit X E ;(0'). Appliquons la formule du corollaire 54 a z = yexp(X). On obtient

L fu(w)eWA(X) = 0 WEW(u)

si ,\ est singulier. La proposition en resulte facilement. •

Nous proposons maintenant une generalisation de la definition 2 d'une famille coherente de modules. Soit V un (D, K)-module. On note AYK

208 M. Du.flo and M. Vel'Yne

l'espace des fonctions A : YK -+ L(V, V) verifiant (en notant r l'action de K et de D dans V) les relations A(kyk' ) = r(k)A(y)r(k' ) pour k E K, k' E K et y E YK, et A(y)r(X) = r(Ad(y)X)A(y) pour y E YK et XED.

On definit un (D, YK )-module comme une paire (A, V) formee d'un (D, K)-module V et d'une fonction A E A yK .

Soit 13 l'ensemble des classes d'equivalence de (D, YK )-modules (A, V) tels que V soit de longueur finie comme D-module, et soit n l'espace vectoriel complexe ayant pour base 6. On note R(;OO(y) l'espace vectoriel complexe obtenu en prenant Ie quotient de n par Ie sous-espace engendre par les elements

1) (cA, V) - c(A, V) si c E IC, (A, V) E 13 2) (A, V) + (A', V) - (A + A', V) si (A, V) E 13, (A', V) E 13 3) (A, V) + (A', V') - (A", V") si (A, V) E 13, (A', V') E 6 (A", V") E 13

et s'il y a une suite exacte courte 0 -+ V -+ V" -+ V' -+ 0 compatible aux actions A, A' et A" .

Si Y = G, on a facilement l'egalite

Soit V un (g, K)-module simple. Soit y E YK • Alors YV est encore un (g, K)-module simple dont la classe ne depend pas de y. On note sa classe <TV.

Lemme 56 Soit (A, V) un (g, YK )-module, de longueur finie comme D­

module. La classe de (A, V) dans R(;OO(y) est nulle si la suite de composition de V comme (g, K)-module ne contient aucun facteur simple fixe par a.

Demonstration. Grace it la condition 3), et en utilisant que Vest de longueur finie comme g-module, on se ramene au cas OU Vest une somme directe de (g, K)-modules simples. On ecrit V = E9l'i ou les l'i sont les composantes isotypiques du D-module V. On ecrit A = 'L,j A,j , OU A'j est un endomorphisme de V nul en dehors de l'i et qui envoie l'i dans Vi, On verifie facilement que chacune des applications Alj appartient it AyK(V). II suffit done de montrer que la classe de (A,j , V) est nulle. Cela resulte de la relation 3) appliquee it la suite exacte

o -+ Vi -+ Vi $ l'i -+ l'i -+ 0,

munie des operateurs 0, A'j et 0 respeetivement. •

Soit >. E ;*(a). Notons Rcoo(yh Ie sous-espace de RcOO(y) engendre par les elements (A, V) E 13 dont tous les sous-quotients simples de V ont

Families coherentes 209

Ie caractere infinitesimal >.. Il resulte des lemmes 56 et 40 que 1'0n a

RCClO(y) = RCClO(Yh· >.EW(a)\J*(a)

On definit de maniere analogue un espace vectoriel Rc(Yc) forme de classes (A, V) ou Vest une representation rationnelle de Ge de dimension finie, et A E Ayc(V). Le caractere, qui a un element (A, V) forme d'une representation rationnelle de Ge et de A E Ayc(V) associe la fonction Tr A, induit une application lineaire de Rc(Yc) dans F(yc)Gc dont on voit facilement, grace a l'analogue du lemme 56, que c'est un isomorphisme. Par ailleurs, Ie produit tensoriel induit une structure d'algebre commutative sur Rc(Yc), isomorphe a F(yc)Gc, et une structure de Rc(Yc)-module sur RC'Xl(Y).

Definition 57 Soit >'0 E ;*(a). Une famille coherente d'elements de RCClO(y) basee sur P(a) + >'0 est une application C : M>'o -+ R(;ClO(y) verifiant les conditions suivantes.

1) Soient V une representation rationnelle de Ge de dimension finie, A E Ayc(V) et £jJ E M>'o. Alors C(Tr(A)¢) est la classe du produit tensoriel de (A, V) et de C(¢).

2) Soit y E YJ. Pour tout>. E P(a) + >'0 on a C(ey ,>.) E RCClO(y»..

On peut montrer qu'un element (A, V) E R(;ClO(y) a un caractere Tr A qui est un element de C-ClO(y)G, et que Ie caractere TrC d'une famille coMrente C comme dans la definition 57 est un famille cohCrente Tr C E

<I>(Y, P(a) + >'0).

2.3 FamilIes s-coherentes et descente.

Nous procooons comme dans Ie paragraphe 1.4. Soit s E Yell. Posons ~ = {I(s), H = G(s) et He = Gc(,;;(s». Comme Ad(s) est semi-simple, He est connexe (tMoreme 38). La sous-algebre ~ de 9 est reductive. On choisit une algebre de Cartan ; de ge telle que; n ~c soit une sous-algebre de Cartan de ~c. Avec les notations du paragraphe 2.1, on a ;(a) =; n ~e et ,;;(s) E YJ. Posons ,;;(s) = ,;;(s)Ja E YJ/Ja . So it v E W(a). L'element v(,;;(s» de YJ/Ja est bien defini et nous noterons, un peu abusivement, V· ,;;(s) un representant de v(,;;(s» dans YJ.

On utilise la decomposition {lc = ~e EI1 (Ad(s) - l){lc pour definir l'application de restriction c~ : S(gc)G -+ S(~c)H. De meme, on considere ;*(a) comme un sous-espace de ~~ et de g~. Soit U un ouvert H-invariant de ~. Soit >. E ;*(a). On note C-ClO(U)f l'espace des fonctions generalisees H-invariantes f dansU qui verifient c~(u)'f = u(>.)f pour tout u E S(gc)G.

210 M. Dufto and M. Veryne

On considere C-oo(U)H comme un F(yc)Ge module pour lequell'action de f E F(yc)Ge est la multiplication par la fonction analytique sur ~ X 1-+

f(K,(s) exp X).

Definition 58 Soient S E Yell et >'0 E ;* (a). Soit U un ouverl H -invariant de~. On appelle famille s-coherente de fonctions generalisees sur U basee sur P(a) +>'0 une application W : MAo -+ C-oo(U)H verifiant les conditions suivantes.

(1) L'application West un morphisme de F(yc)Gc-modules. (2) Soit y E YJ. Pour tout>. E P(a) + >'0 on a w(ey,A) E C-OO(U)f.

On note iP(U, P(a) + >'0, s) l'espace de ces familIes coMrentes.

Remarque 59 Dans le cas ou Y = G, on retrouve les familles coherentes de la definition 21 en posant F(>.) = W(el,A)'

Notons 'Y~ Z(gc) -+ S(~c)H l'homomorphisme compose de l'isomorphisme d'Harish-Chandra Z(gc) '" S(gc)G et de c". Soit D(~c) l'algebre des operateurs differentiels de Chol(~c). II existe un homomorphisme d'algebres de DYe dans V(~c), encore note 'Y", qui cOIncide avec 'Y" sur Z(gc) , et tel que, pour f E F(Yc)Ge et X E ~c, on ait 'Y,,(f)(X) = f(K,(s) exp X) -ceci resulte de la proposition 61 ci-dessous-. Rappelons l'homomorphisme h;" de DYe dans V(YJ / Ju ). Les conditions (1) et (2) de la definition 58 peuvent encore s'ecrire

(53) w(h;,,(u)· ¢) = 'Y,,(u) . w(¢)

pour tout u E DYe et ¢ E MAO.

Soit >'0 un element regulier de j(a)*. II resulte du lemme 44 qu'une famille coMrente W E iP(U, P(a) + >'0, s) est determinee par la fonction generalisee w(ey,AO) E C-OO(U)~. Reciproquement, si U est completement invariant, on peut demontrer que pour tout element f E C-OO(U)~ il existe une famille coMrente W E iP(U, P(a) + >'0, s) telJe que l'on ait w(ey,AO) = f.

Donnons quelques exemples. Supposons pour commencer que S soit regulier dans Y. On a done

j(a) = k Soient >'0 E ;*(a), v E W(a) et y E YJ • Pour J.L E P(a) + >'0, posons

wv(eY,I'+AO) = eY,I'(v-1 • K,(s))ev(l'+>'o).

On peut encore ecrire ceci sous la forme

wv(¢ey,>.o)(X) = ¢(v-1 • (,.,;(s)eX))eVAO(X)

pour tout ¢ E F(YJ/Ju) et tout X E~. On verifie immediatement que l'on a Wv E iP(~, P(a) + >'0, s) et l'on obtient Ie lemme suivant.

Families cohirentes 211

Lemme 60 Soit s E Yell un element regulier. Les Wv (v E W(a}) forment un base de 4>(~, P(a} + AO, s}.

Comme au paragraphe 1.4, on obtient des familIes s-coherentes par la methode de descente. Soit a > o. Rappelons que nous avons defini 1'0uvert elliptique ~a de~. Comme en (25) on note W(s, a} Ie sous-ensemble

W(s,a} = {gsexp(X}g-1,g E G,X E ~a}

de Y. On definit Ie nombre f(S} > 0 comme dans Ie paragraphe 1.4. Supposons que 1'0n ait 0 < a ::; f(S}. Soit f E C-OO(W(s, a}}, on definit encore Es(f} E C-OO(~a} par la formule (28). Dans ce cadre l'analogue de la proposition 25 est encore valable (Bouaziz [5] et Harish-Chandra [12]).

Proposition 61 Supposons que l'on ait 0 < a ::; f(S}. Pour tout u E Z(gc) et tout f E C-OO(W(s,a»G, on a Es(u· f) = 'Y~(u}· Es(f}.

Soit V un ouvert eUiptique G-invariant de Y et soit s E V n Yell. On definit Ie nombre fV(S} > 0 comme dans Ie paragraphe 1.4. Soit AO E ;*(a). Soit W E 4>(V, P(a} + Ao}. Soit a tel que I'on ait 0 < a ::; fV(S}. Pour tout ¢ E M>'o on pose

II resulte de la proposition 61 que 1'0n a

(54)

Donnons encore l'exemple des familIes principales. On suppose que l'algebre derivee I de ~ est deployee. On choisit une algebre de Cartan a de ~ telle que a n I soit deployee, et on choisit j telle que 1'0n ait j(a) = Ge, de sorte que W~ laisse stable a. Pour tout ¢ E C-OO(a), on definit la fonction generalisee H-invariante sur ~ notee B!rinc(¢) par la formule (29). Soient AO E ;*(a) et v E W~ \W(a). Pour A E P(a) posons

(55) ,T, ( ) ('-1 (»B~ (1)().+>'o» ':Ie" v eY,>'+>'Q = ey,>. v • I'i, S prine e ,

ou vest un representant de v dans W(a). On verifie immediatement que 1'0n a Wv E q>(~, P(a) + AO, s).

On definit Ie sous-ensemble C~ comme en (31). Done C~ est l'adMrence de l'ensemble des elements de ~ conjugues d'un element de a.

La proposition suivante generalise Ie lemme 60 et Ie corollaire 28 et se demontre de la meme maniere.

212 M. Dujlo and M. Veryne

Proposition 62 Soit s E Yell. On suppose que l'algcbre db'ivee de ~ est deployee. Soit a> O. Soit W E ~(~a, P(a) + AO, s). On suppose qu'il existe un element regulier A E P(a)+Ao tel que Ie support de w(ey,A) soit contenu dans c~n~a. Alors west combinaison lineaire des wv , avec v E W~ \W(a).

Enon<;ons pour terminer ce paragraphe Ie resultat que nollS avons en vue. Soit s E YK • Nous employons les notations ~, H, etc ... ci-dessllS. Soit U un ouvert de~. Toute fonction generalisee f E C-OO (U)lfin admet une restriction fl~nt at n ~ n u. Le theoreme ci-dessous sera demontre dans Ie paragraphe suivant.

Theoreme 63 Soient s E YK , 0 < a $ e::(s), AO E j"(a) et W E ~(g(s)a,P(a) + AO,S). On suppose que l'on a w(¢»lg(s)nt = 0 pour tout ¢> E MAo. Alors on a W = O.

2.4 Demonstration des theoremes d'unicite.

Soit s E Yell. Nous employons les notations ~ = g(s), etc ... du paragraphe precooent. Soit U un ouvert H-invariant elliptique de ~ et soit S E Uell . On pose s' = seS • C'est un element elliptique de Y. On suppose que l'on a g(s') = g(s) n g(S). On choisit 0 < a $ e::(S) tel que W(~, S, a) C U. Rappelons la definition E~,s (formule (35».

Soient AO E ;*(a) et W E ~(U, P(a) + AO, s). Pour tout ¢> E MAo, on pose (56) (Rs,sW)(¢» = E~,s(W(¢>)). Comme dans Ie lemme 32, on a

(57) Rs,sW E ~(g(S')a, P(a) + AO, s').

Demonstration du theoreme 63. On la fait par recurrence sur la dimension de g(s) comme celIe du theoreme 30 donnee au paragraphe 1.5. L'hypothese de recurrence et la formule (57) permettent de se ramener a la situation suivante.

Soit s E YK • Posons ~ = g(s), notons ( l'algebre derivee de g(s) et Cs

l'ensemble des X E g(s)a tels que S(X) appartienne au centre 3 de g(s). On suppose que (est deployee et que Ie support des fonctions generalisees w(¢» est contenu dans Cs pour tout ¢> E MAO. Soient Cv E IC (v E W~\W(a»

Families coherentes 213

les constantes telles que ron ait W = EVEW~\W(O") cvwv (voir la proposition 62). Soit y E YJ . 11 resulte de la formule (32) que, pour tout>. E P{a), la restriction de w{ey,.\+>.o) at n ( est egale a EVEW~\W(O") Cvey,.x{v-1 . /l,{s) )6/rt .

L'hypothese du theoreme entraine donc que ron a

(58) L cvey,.x{v-1 • /l,{s)) = 0 vEW~\W(O")

pour tout>. E P{a). Comme les points v-1{/l,{s)JO") de YJ/JO" sont deux a deux distincts quand v parcourt W~ \W{a) (point c) du lemme 39), les constantes Cv sont nulles. •

Demonstration du theoreme 46. Compte tenu du theoreme 63, elle est analogue it celle du theoreme 6 donnee au paragraphe 1.5. •

References

[1] M. F. ATIYAH ET R. BOTT. A Lefschetz fixed-point formula for elliptic differential operators. Bull. Amer. Math. Soc. , 72 (1966), 245-250.

[2] M. F. ATIYAH ET R. BOTT. A Lefschetz fixed-point formula for elliptic complexes: II. Ann. of Math., 88 (1968), 451-491.

[3] G. BARBANQON ET M. RAIs. Sur Ie theoreme de Hilbert differentiable pour les groupes lineaires finis (d'apres E. Noether). Ann. Scient. Ec. Norm. Sup., 16 (1983), 355-373.

[4] R. BOTT. Homogeneous vector bundles. Annals of Math., 48 (1957), 203-248.

[5] A. BOUAZIZ. Sur les caracteres des groupes de Lie reductifs non connexes. J. of Functional Analysis, 70 (1987), 1-79.

[6] N. BOURBAKI. Groupes et algebres de Lie, Chap. 4, 5 et 6. Masson, Paris, 1981.

[7] M. DUFLO ET M. VERGNE. Cohomologie equivariante et descente. Asterisque, 215 (1993), 5-108.

214 M. Dufto and M. Ve1Yne

[8] M. DUFLO ET M. VERGNE. Un theoreme d'unicite pour les familles coherentes sur les groupes semi-simples. C. R. Acad. Sci. Paris, 317 (1993), 1001-1006.

[9] M. DUFLO, G. HECKMAN ET M. VERGNE. Projection d'orbites, formule de Kirillov et formule de Blattner. Mem. Soc. Math. Fr., 15 (1984), 65-128.

[10] HARISH-CHANDRA. Differential operators on a semisimple Lie algebra. Amer. J. Math. , 79 (1957), 87-120.

[11] HARISH-CHANDRA. Invariant eigendistributions on a semisimple Lie algebra. lnst. Hautes Etudes Sci. Publ. Math. , 27 (1965), 5-54.

[12] HARISH-CHANDRA. Invariant Eigendistributions on a Semisimple Lie Group. Trans. Amer. Math. Soc., 119 (1965), 457-508.

[13] HARISH-CHANDRA. Discrete series for semi-simple Lie groups I. Acta Math., 113 (1965), 241-318.

[14] R. HOTTA ET M. KASHIWARA. The Invariant Holonomic System in a Semisimple Lie Algebra. Inventiones Mathematicre, 75 (1984),327-358.

[15] B. KOSTANT. Lie algebra cohomology and the Borel-Weil theorem. Annals of Math., 74 (1961), 329-387.

[16] T. LEVASSEUR ET J. T. STAFFORD. Invariant differential operators and an homomorphism of Harish-Chandra. Preprint 1993

[17] W. SCHMID. Two characters identities for semisimple Lie groups. Springer Lecture Notes in Math., 787 (1976), 196-225.

[18] M. A. SEMENOV-TIAN-SHANSKY. Harmonic analysis on symmetric spaces of nonpositive curvature and scattering theory. SOy. Math. lzv., 40 (1976), 562-592.

[19] R. STEINBERG. Endomorphisms of linear algebraic groups. Mem. Amer. Math. Soc. 80, 1968.

Families coherentes 215

[20] V. S. VARADARAJAN. Harmonic analysis on real reductive groups. Springer Lecture Notes in Math. 576, 1977.

[21] M. VERGNE. Geometric quantization and equivariant cohomology. A paraitre dans les proceedings du Congres Europeen, Paris 1992.

[22] D. A. VOGAN. Irreducible characters of semis imp Ie Lie groups II. The Kazhdan-Lusztig conjectures. Duke Math. J., 46 (1979), 805-859.

[23] D. A. VOGAN. Representations of real reductive Lie groups. Birkhiiuser, Boston, 1981.

[24] D. A. VOGAN. Irreducibility of discrete series representations for semi­simple symmetric spaces. Advanced Studies in Pure Math., 14 (1988), 191-221.

[25] N. R. WALLACH. Real reductive groups I. Academic Press, New-York, 1988.

[26] N. R. WALLACH. Invariant differential operators on a reductive Lie algebra and Weyl group representations. Journal of the Amer. Math. Soc. , 6 (1993), 779-816.

[27] G. ZUCKERMAN. Coherent translation of characters of semisimple Lie groups. Proceedings of the International Congress, Helsinki 1978, 721-724

Michel Duflo, Universite Paris 7-Denis Diderot et URA 748 du C.N.R.S.

Michele Vergne, E.N.S. et UA 762 du C.N.R.S.

Received January 12, 1994

The Differential Geometry of Fedosov's Quantization

Claudio Emmrich1 and Alan Weinstein2

Abstract

B. Fedosov has given a simple and very natural construction of a deformation quantization for any symplectic manifold, using a flat connection on the bundle of formal Weyl algebras associated to the tangent bundle of a symplectic manifold. The connection is obtained by affinizing, nonlinearizing, and iteratively flattening a given torsion free symplectic connection. In this paper, a classical analog of Fe­dosov's operations on connections is analyzed and shown to produce the usual exponential mapping of a linear connection on an ordinary manifold. A symplectic version is also analyzed. Finally, some re­marks are made on the implications for deformation quantization of Fedosov's index theorem on general symplectic manifolds.

1. Introduction

In a remarkable paper [10], B. Fedosov presented a simple and very natural construction3 of a deformation quantization for any symplectic manifold. The construction begins with a linear symplectic connection on the tangent bundle of a manifold and proceeds by iteration to produce a flat connection on the associated bundle of formal Weyl algebras. The aim of this paper is to give some geometric insight into Fedosov's construction by explor­ing some of its purely classical analogs. Our basic idea is that Fedosov's quantization procedure involves a sort of "quantum exponential mapping."

The paper is divided into two parts. The first part (Sections 2 through 4) is mostly discursive and follows fairly closely the lecture given by the second author at the conference in honor of Bert Kostant's 65th birthday. The second part (Sections 5 through 8), more technical and logically self­contained, develops the proofs of some of the statements in the first part.

lResearch supported by Deutsche Forschungsgemeinschaft, Az.: Em 47/1-1. 2Research partially supported by NSF Grants DMS-90-01089 and DMS-93-

09653. :1 Actually, this construction appeared in several of his earlier papers, such as

[7) and [8).

218 C. Emmrich and A. Weinstein

2. Deformation quantization and symplectic connections

The basic deformation quantization problem is to put a noncommutative (associative) product structure on the formal power series ring A[[h.J] over a commutative Poisson algebra A. The physical origins of this particular formulation of the classical limit are obscure to this author (see Section 16.23 in [3) for an occurrence in a textbook), but the origins of its recent intensive study seem to lie in the work of Berezin [2) and Bayen et al [1). We refer to the paper of Flato and Sternheimer in this volume [11) for a general discussion of deformation quantization, and to [18) for a report on Fedosov's work.

The deformed product, which is determined by the product of elements of A and can be written in the form

1 *Ii 9 = L h.j Bj(f,g) j

for bilinear operators Bj on A, is assumed to start with the original com­mutative product followed by the Poisson bracket, i.e.,

1 *Ii 9 = Ig + (ih./2){f,g} + ... It was already noted in [1) that, when A is the algebra of Coo functions

on a symplectic manifold P, the term of order h.2 in the deformed product is closely related to a torsion-free symplectic connection on P. If there exists such a connection which is fiat, one can immediately write down a solution of the deformation quantization problem. In fact, P is then covered by coordinate charts for which the transition maps are affine symplectic trans­formations, which leave invariant the Moyal product [1) on coo (lR2n)[[h.J]. (The algebra with this product is often called the (smooth) Weyl Algebra.)

When P does not admit a flat symplectic connection (or where such a connection cannot be invariant under an important symmetry group, such as in the case of the hyperbolic plane with its PSL(2, JR) symmetry), it is necessary to "piece together" quantizations like the one above in a more elaborate way. The first such construction to work for all symplectic manifolds was that of DeWilde and Lecomte [5) who actually piece together quantizations on individual coordinate systems with nonlinear transition maps.

A more geometric formulation of the "piecing construction" was given by Maeda, Omori, and Yoshioka in [15]. Their construction has important elements in common with that of Fedosov, so we will describe it in some detail here.

For any symplectic manifold P, we consider its tangent bundle TP with the Poisson structure for which each fibre is a symplectic leaf, carrying the translation invariant symplectic 2-form associated with its structure

The Differential Geometry of Fedosov's Quantization 219

as a symplectic vector space. It is very simple to quantize this Poisson manifold - we simply quantize each fibre by the Moyal product, so that the noncommutative algebra COO(TM)[[lilJ may be identified with the set of "smooth" sections of a bundle whose fibre over each point p of P is the smooth Weyl algebra of the tangent space. Geometrically, we think of COO(TM)[[lilJ as the "space of functions on the quantized tangent bundle."

To quantize P itself, we may try to embed COO(P)[[lilJ as a subalgebra of this noncommutative COO(TP)[[lilJ. The embedding onto the functions which are constant on fibres of T P does not work, since this subalgebra is commutative, so we need a different embedding. In geometric terms, we are trying to replace the usual projection of T P to P by a different one.

To see what this new projection should look like, we may examine the simplest case, where P is a symplectic vector space. Denoting by x some linear coordinates on P and (x, y) the corresponding coordinates on T P , the multiplication on the quantized T P is just the "Moyal product in y, with x as a parameter." If we identify each function f(x) on P with the function Lf(x,y) = f(x + y) on TP, then taking the fibrewise Moyal product of Lf and Lg reproduces the "Moyal product in x" of f and g. The operator L is just pullback by the map (x, y) 1-+ (x + y), which is nothing but the exponential mapping of P with its flat affine structure. (Here, the deformation parameter Ii just comes along for the ride.) It is not hard to see that the same idea works for any symplectic manifold with a flat, torsion-free symplectic connection: pulling back functions from P to T P by the exponential map allows one to pull back the multiplication on the quantized tangent bundle to get the aforementioned Moyal-type deformation quantization of P.

At this point, one might object that the connection on P might not be complete, in which case its exponential map is not globally defined. In this case, it suffices to restrict the fibrewise Moyal product to the open subset of T P which is the domain of the exponential map.

When the connection on P has nonvanishing torsion or curvature, the situation is much worse, in that the set of functions pulled back from P by the exponential map is not closed under the multiplication on the quan­tized T P. Two remedies for this problem have been used in the literature. We may describe them roughly as follows. In [15], the local exponential mappings coming from a covering of P by Darboux coordinate systems are "patched together" to produce a new projection which is no longer (at least not manifestly) the exponential map of a connection. This new projection is built at the "quantum level": that is, what is actually constructed is a patching together of modifications of the local pullback mappings from functions on the base to subalgebras of local sections of the bundle of Weyl algebras. In fact, this description is slightly oversimplified. It is not the bundle of smooth Weyl algebras but rather the bundle of formal Weyl alge-

220 c. Emmrich and A. Weinstein

bras (formal power series around zero of functions on the tangent spaces) which must be used. Also, the parameter n now plays an essential role, in that the pullback of a function on the base is no longer independent of n.

Fedosov [10), on the other hand, beginning with a linear symplectic connection having zero torsion but nonvanishing curvature, lifts it to a connection on the bundle of formal Weyl algebras and then modifies the connection on this bundle, with structure Lie algebra the inner derivations, to make it flat. He then shows how the space of parallel sections of this bundle, which is clearly a subalgebra of the algebra of all sections, may be identified with the space Coo{P)[[n)).

The idea which we propose in this paper is that Fedosov's construction can also be interpreted on the level of spaces. The parallel sections of the bundle of formal Weyl algebras are defined by the vanishing of covariant differentiation operators, which are derivations on the space of sections. When the space of sections is considered as the space of functions on the quantized TP (more precisely, on a formal neighborhood of the zero section in the quantized T P), these derivations may be thought of as vector fields, determining a distribution on TP which is transverse to the fibres, i.e. an Ehresmann connection, on T P. The flatness of Fedosov's connection is equivalent to the involutivity of the distribution, or the flatness of the Ehresmann connection. Thus, the parallel sections of the formal Weyl algebra bundle are interpreted as functions on the quantized T P which are constant along the leaves of the quantum foliation which is tangent to the quantum Ehresmann connection.

To pull back a function on P, we first identify it with a function on the zero section, then extend it to a function on T P constant along leaves. This is possible if the Ehresmann connection is not tangent to the zero section of T P, but is rather transverse to the zero section, so that each leaf intersects the zero section in a unique point. To achieve this, even if the initial linear connection is flat, we must modify it by the addition of a "translation" term to obtain a connection with affine structure group.

The somewhat mysterious fact that a parallel section for the Fedosov connection is not determined by its value at a single point can also be attributed to the addition of the translational term, which changes the way in which the covariant differential operator behaves with respect to the grading in the formal Weyl algebra. In other terms, the fact is related to the non-determination of a Coo function by its Taylor series at one point. An analogous situation occurs on the bundle of infinite jets of real-valued functions on P, which carries a "flat connection" for which the parallel sections are the infinite jets of functions on P. In a sense, what Fedosov's construction does is to identify this bundle of infinite jets of functions on P with the bundle of formal functions on the fibres of T P and then to pull back the natural flat connection from the former bundle to the latter.

The Differential Geometry of Fedosov's Quantization 221

There is another way to see the necessity of "affinizing" the structure group of the connection. If a nonlinear flat connection on T M has the zero section as a parallel section, then its linearization at the origin would be a flat linear connection. There are, in general, obstructions to the existence of such a connection; by allowing the zero vectors to move, we bypass these obstructions.

3. Classical analogs

In the previous section, we used geometric language to give an intuitive pic­ture of the constructions on quantized formal power series in [10]. Now we will turn the tables and apply Fedosov's formal constructions in purely clas­sical settings. A more detailed version of the following remarks is contained in the second part of this paper, beginning with Section 5.

Suppose that we are simply given a differentiable manifold P with a torsion free linear connection on T P. This connection may be lifted to a connection on the associated bundle of algebras of formal power series on the tangent spaces of P. If we extend the structure group of this connection from the Lie algebra of linear vector fields on lRm (m being the dimension of P) to the Lie algebra of formal vector fields, then Fedosov's iteration method can be used to produce, in a canonical way, a flat connection on this bundle for which the corresponding Ehresmann connection is transverse to the zero section. The leaves of the foliation given by the flat Ehresmann connection are the fibres of a "mapping" from a formal neighborhood of the zero section in TP to P. We denote this formal mapping by EXP.

On the other hand, there is another well known mapping from T P to P determined by a connection, namely the usual exponential mapping exp . What is the relation between these two mappings? Since both are produced in a canonical manner from the connection, it is natural to guess that they are equal. In fact this is true, as we prove below. The structure of the proof is of some interest. We first prove that, for a real-analytic connection, the flat connection given by Fedosov's iteration is given by convergent power series, so it actually defines a flat real-analytic Ehresmann connection on a neighborhood of the zero section. In this case, we prove by following geodesics that EXP and exp coincide. Next, we show that EXP can be defined in the purely formal context by "extension by continuity," using the fact that the convergent power series (already the polynomials) are dense in all the power series with respect to the uniformity for which two power series are close if all their coefficients up to some high degree are equal. Finally, we conclude that EXP and exp are equal since they agree on a dense subset. Since all the constructions are local, they then apply on any manifold, without any reference to a real-analytic structure.

In a sense, we have shown that the iterative procedure used by Fedosov

222 C. Emmrich and A. Weinstein

to "flatten" a connection, when applied to ordinary manifold, is simply another way of constructing the usual exponential mapping. It is in this sense that we are tempted to say that his quantization procedure involves the pullback of functions from a symplectic manifold P to its quantized T P by quantization of the exponential mapping of a given symplectic connec­tion. Before taking this leap, though, we must be more careful, because the exponential mapping of a symplectic linear connection on a symplectic manifold does NOT in general define symplectic mappings from the tangent spaces with their constant symplectic structures to the manifold with its given symplectic structure.

To produce from a symplectic torsion-free linear connection on TP a symplectic nonlinear connection (on a formal neighborhood of the zero sec­tion), we must change the structure Lie algebra from the algebra of formal vector fields to the algebra of formal symplectic vector fields. In fact, with a view toward quantization, it is useful to use instead the one-dimensional extension of the latter algebra given by the infinite jets of functions, with the Poisson bracket Lie algebra structure. As is well known, this Lie alge­bra acts on itself by derivations of both the multiplication and the Poisson bracket. Now the iterative procedure can be applied again in this symplec­tic classical context to produce (formal) symplectic "exponential" mappings from the fibres ofTPto P. It would be interesting to have a more geomet­ric construction of these mappings. It appears that they may be the same as those obtained by starting with the ordinary exponential mappings and then correcting these by using the deformation-method proof of Darboux's theorem in [16], using the linear structure in the fibres of TP.

Recently, we have noticed that our construction of the symplectic "ex­ponential" mapping is precisely the limit as Ii --+ 0 of Fedosov's. Conse­quently, his construction may be seen as giving "quantum corrections" to a purely geometric one.

4. Rigidity of quantization and Fedosov's index theorem

In the sections above, we have concentrated on Fedosov's construction of a deformation quantization. This construction was, however, merely the be­ginning of Fedosov's work, one of whose points of culmination is an index theorem for arbitrary symplectic manifolds [8]. This theorem is a gener­alization of the Atiyah-Singer theorem, to which it can be reduced (not without some effort) when the symplectic manifold is a cotangent bundle.

In this section, we wish to comment on the role which Fedosov's index theorem might play in resolving a fundamental question in the general theory of deformation quantization. The condition

f *1i 9 = fg + (ili/2){f,g} + ...

The Differential Geometry of Fedosov's Quantization 223

which describes the n dependence of the product *Ii says nothing about how the product behaves after the deformation parameter "leaves the first­order neighborhood of zero." This weak condition seems to contrast with the rigidity implied by the special role of rational values of q = eili in the theory of quantum groups. The question thus arises as to whether the dependence on n should be "rigidified" by some supplementary condition ( s).

A possible source of such rigidity is suggested in [17], where in the spe­cial case of quantizations of Moyal type on tori with translation-invariant Poisson structures it is shown that the Schwartz kernel of the bilinear mul­tiplication operator *Ii satisfies a Schrodinger equation in which Planck's constant plays the role of time, and the hamiltonian operator is the Poisson structure extended by translation invariance to become a differential opera­tor on P x P. Although this kind of evolution equation for quantization may possibly extend to other situations where the notion of translation makes sense (e.g. on Lie groups), it is not at all clear how to extend it to general symplectic manifolds.

Another kind of rigidity in the deformation parameter appears in the geometric quantization of Kahler manifolds by sections of holomorphic sec­tions of line bundles (and the extension of this quantization to general symplectic manifolds by the method of Toeplitz operators in [4]). The de­formation parameter n then occurs as a unit by which the cohomology class of the symplectic structure (which in physical examples carries the units of action) is divided to obtain a cohomology class with numerical values which, when integral, is the first Chern class of a complex line bundle. When this class is integral for n = ho, then as n runs over the set of val­ues holk, for k a sufficiently large integer, the dimension of the space of holomorphic sections is given according to the Riemann-Roch theorem by a polynomial in k of degree equal to half the dimension of the symplectic manifold, which is completely determined by the cohomology class of the symplectic structure and the total Chern class of the tangent bundle for an almost complex structure compatible with the symplectic structure. This behavior shows that geometric quantization has a very rigid behavior with respect to n.

The index theory of Fedosov [8] provides an analog in deformation quantization of the rigidity described in the preceding paragraph. Accord­ing to this theory, when his deformation quantization is extended from scalar functions to matrix-valued functions on P, a notion of "abstract el­liptic operator" can be defined. When a suitable trace is introduced, the index of such an operator can be defined. Fedosov's index theorem then expresses this index as a polynomial in (lin) which is completely deter­mined by the Chern character of a certain vector bundle over P associated to the operator, the A class of the tangent bundle of P (with an almost complex structure), and the cohomology class of the symplectic structure.

224 c. Emmrich and A. Weinstein

Once again, this formula suggests that some quantizations (for which this formula is true) are "better" than others which might have the same first­order behavior with respect to n.

We refer to [9] and [14] for further discussion of the classification of deformation quantizations and its relation to index theory.

5. Construction of formal flat connections

In this section we study in detail the classical analog of the flat connection D constructed in [10] for the quantum case. In this classical setting the Weyl algebra bundle is replaced by A = :F(TP), the algebra of smooth functions on TP, or, if one restricts oneself to the study of formal power series on the fibres, by A = r(UPEP..1000(TpP,JR)), where ..1000(TpP,JR) denotes the set of oo-jets at 0 of real valued functions on TpP. The product on A in the non-symplectic setting is either pointwise multiplication for A = :F(TP), or, for A = r(UPEP..1000(TpP,JR)), the mapping induced by it on the set of oo-jets, respectively. In the symplectic case studied in Section 8 this product is supplemented by the fibrewise Poisson bracket (Le., the Poisson bracket on the fibres of TP induced by the symplectic form w on P), to produce a Poisson algebra structure.

Let A(P) be the set of forms on P. The contraction (ixa)(·) is defined as a(X,·) for a vector X and a form a. As in [10], we can define operators 8 and 8-1 on A ® A(P), given in local coordinates (xi) on P and induced coordinates (xi,yi) on TP by:

and

. () {j = dx' 1\ -() .

y'

£-1 1. u a = -- Zyi {} a

p+q o;r

(1.1)

(1.2)

if a is a q-form that is homogenous of degree pin y, p+q:f. 0, and 8- 1a = 0 if p = q = O.

Let Vinv(TP) denote the set of invariant vertical vector fields, Le., of vertical lifts of vector fields on P. We can define the operators 8 and 8-1 on forms with values in the 'formal vertical vector fields' A®Vinv(TP)®A(P) as well, by letting them act trivially on vinv(TP). With this definition, the "Hodge decomposition"

(1.3)

still holds for a E A®Vinv(TP)®A(P). Here, aoo denotes the homogeneous part of a which is a zero-form of degree 0 in the vertical coordinates (yi); Le. it is simply an element of vinv(TP).

The Differential Geometry of Fedosov's Quantization 225

We would like to construct from a given linear, torsion-free, nonflat con­nection 8 a flat, nonlinear connection on T P with structure group Diff{JRn) (or the symplectomorphism group of JRn for even n in the case of the Poisson algebra studied in Section 8), or equivalently, a flat Ehresmann connection on TP. Locally, such a connection is represented by a one-form X tot on P with values in the vertical vector fields, yielding for the local expression for the curvature n, i.e., the obstruction to the integrability of the horizontal distribution defining the connection:

n = dXtot + 1/2[Xtot ~ Xtotl

where [. ~ .] acts as the Lie bracket on the vector part and as the exterior product on the form part of X. A connection induces a covariant derivative D on elements of A, which is given in local coordinates by:

By analogy with the quantum case in [10], X tot should be locally of the form

8 . . k 8 I X tot = --8 .dx· -rkl{x)y -8 .dx +X,

y' y' (1.4)

where r~,{x) are the Christoffel symbols of the connection 8, and X is at least of second degree in y.

The addition of the term a~. dx i , which corresponds to the operator 8, corresponds to the transition from the linear connection 8 to an affine connection with connection form Xtot - X by the addition of the solder form [12]. The vanishing of the torsion for 8 implies that the "translational" part of the curvature of the affine connection is zero.

For instance, if the connection 8 is flat, we may choose X == 0 in (1.4) to get a flat connection again. In this case, the original horizontal distribution corresponding to 8 is already integrable. For P = JRn with the canonical flat connection, the leaves of the horizontal distribution correponding to 8 are affine subspaces JRn x {v} C JR2n parallel to the zero section Z = JRn x {O} ~ P. This foliation is rotated by the addition of the solder form in such a way that the leaf through (x, v) intersects the zero section in (x + v, 0) for arbitrary (x, v) E JR2n (Figure 1).

226 c. Emmrich and A. Weinstein

Figure 1. Addition of the solder form to a flat linear connection "rotates" the parallel sections (from dotted to solid lines) so that they become transversal to the zero section.

In the general, non-flat case one may interpret the ansatz (1.4) as first going over from the structure group GI(n, JR) with n = dim(P) to the affine group GI(n, JR) <lJRn in order to get a horizontal distribution transversal to the zero section, and then deforming it in order to get a flat nonlinear con­nection, which has an integrable horizontal distribution and hence defines a foliation.

Although the transition from a linear connection to an affine connection is canonical, one could in principle take any multiple of 8 in order to define a generalized affine connection in the sense of [121, i.e., a connection on the GI(n, JR) <l JRn-bundle of affine frames. However, the choice made in the ansatz is not only geometrically natural, but also distinguished by the validity of Theorem 1.3 and Lemma 1.4 below.

The requirement of the vanishing of the curvature yields the condition:

o = n = R - 8X + ax + 1/2[X ~ XI, (1.5)

where ax ~ dX - [ril(X)yk a~.dxl ~ XI, and R denotes the curvature

tensor corresponding to a. The term [a~J dxj ~ ril(X)yk ivrdxll which

one would expect to appear in (1.5) is equal to ril(x)dxk 1\ dxl a~" which vanishes because the connection is torsion-free.

We first try to find a formal power series in y for the solution of Equa­tion (1.5). Solutions of this equation are not unique; however, we will show that they are in one-to-one correspondence to formal vertical vector fields a E A® vinv(T P) of degree at least 3 in y via the 'normalization condition'

8-1 X = a. (1.6)

We will see in Section 6 that 8-1 X is just the X-dependent term that appears in the equation defining autoparallel curves. Hence, the case a = 0

The Differential Geometry of Fedosov's Quantization 227

will prove to be special in so far as the term involving X does not change autoparallel curves: in general 15-1 X can be interpreted as an acceleration term for autoparallel curves.

Therefore it might appear natural to admit only the special case a = O. However, we will see in Section 8 that the boundary condition 15-1 X is not appropriate in the symplectic case. (It must be replaced by the condition 15-1r = 0 for the corresponding hamiltonian function r.) Hence, we will admit arbitrary values of a in the non-symplectic case as well.

Using (1.3) and the vanishing of Xoo (since X is a one-form), we get:

X = 15a + t5-1{R + 8X + 1/2[X ~ X]), (1.7)

which must be satisfied by any solution to (1.5) and (1.6). As in [10], this equation may be solved by iteration, yielding a unique power series for X. In spite of the more general normalization condition (1.6), the argument in [10] (also see the "topological lemma" in [6]) showing that the X found in this way is indeed a solution to (1.5) is still valid: defining A as the right hand side of equation (1.5) with the recursively constructed X, we may check that A fulfills the equation t5A = 8A + [X, A] and the condition t5-1 A = 0 and hence vanishes identically.

Hence, we have proved the following theorem

Theorem 1.1 For each a E A ® Vinv{TP) at least of degree 3 in y there is a unique formal flat connection, i.e., a solution to (1.5), satisfying the normalization condition (1.6).

6. Analytic connections and the "exponential map"

In the last section we constructed a formal flat connection from a given linear, torsion-free connection. In this section we will show that this formal power series is convergent in a tubular neighborhood U of the zero section if the manifold M, the connection 8, and the element a in the normalization condition are real analytic. The proof uses the method of majorants, as discussed in any standard reference on real analytic functions, such as [13].

Theorem 1.2 Let P be a real analytic manifold, 8 an analytic linear con­nection, and a E :F(T P) an analytic function such that a and its derivatives up to order two in the direction of the fibres vanish on the zero section Z. Then the formal power series which is the solution of (1.7) obtained by iter­ation is convergent in a neighborhood U of Z and hence defines an analytic Ehresmann connection on U c T P.

Proof: Choose coordinates (xi) on P and induced coordinates (xi, yi) on TP. We define a linear operator T on power series (divisible by y) in two

228 C. Emmrich and A. Weinstein

variables (x, y) by its action on monomials in y:

We will show:

T k A.( ) def 1 k d A.( ) Y 'I' X = -y -'I' X k dx

1. The power series of the components Xj of X = Xj 8~' dxi are ma­jorized by fCE Xi, E yi), where f(x, y) is a power series with no terms of order zero in y which is a solution of:

y2 y f(x, y) = A( (1 _ .Bx)(1- .By) +yTf(x, y)+ 1 _ .Bxf(x, y)+ f(x, y)2).

(1.8) for sufficiently large A > 0 and .B > O.

2. f(x, y) is convergent in a neighborhood of (0,0).

Hence, X is analytic. To show 1, we first observe that Equation (1.8) has a unique solution

containing no term of order zero in y, which can be computed by iteration. We say that f majorizes Xj up to order k in y iff the absolute value of

the coefficient of any monomial xl-'yV in Xj for arbitrary multiindices p, v with Ivl ~ k is not larger than the respective coefficient in f(E Xi, E yi).

Both Xj and f(E Xi, E yi) contain no terms of order zero or one in y. As the coefficients of any analytic function p(z) = EI-'ENN Ol-'zl-' in variables (Z1, .. . , zN) have to satisfy 101-'1 < ABII-'I for some constants A, B, the function p is majorized by a/(1 - .B(z1 + '" + zN» for some a,.B > O. From this general statement it follows that there are a,.B such that the components of {ja + {j-1R are majorized by ay2(1 - .Bx)-1(1 - .By)-1 , and the Christoffel symbols r~, are majorized by a(1 - .Bx)-1, where x = Exi,y = Eyi. Hence, from equations (1.7) and (1.8) it is obvious that for A > a the power series f(E Xi, E yi) majorizes all Xj up to order 2 in y.

Now assume that f(E xi, E yi) majorizes all Xj up to some order kEN. Then, by construction ofT, the components of {j-1dX are majorized up to order (k+ 1) by 2NyTf(x,y), where N = dimP. The components of 1/2[X t;. Xl are majorized by 2N2 f(x, y) d~f(x, y) up to order k. Thanks to the factor 1/(p + q) in the definition (1.2) of {j-1, the components of {j-11/2[X t;. Xl are majorized up to order (k + 1) by

4N31Y f(x, fJ) d~f(X, fJ)dfJ = 2N3 f(x, y)2,

Finally, the components of [r~lyk 8~' dxk t;. Xl are majorized up to order

k by N 2a/(1 - .Bx)(f + y d~f) and {j-1 [r~lyk 8~' dxk t;. Xl is majorized up

The Differential Geometry of Fedosov's Quantization 229

to order (k + 1) by

2N31Y 0:/(1 - {3x)(f(x, y) + y :UI(x, y»dy = 2N30:/(1 - {3x)yl

Hence, by equations (1.7), (1.8), X; is majorized up to order (k + 1) in y

by I(E Xi, E yi), and 1 is shown. To prove 2 we show, again by induction on the degree in y, that I is

majorized by the solution g(x,y) which has no terms of degree zero or one in y of the purely algebraic equation

- y2 Y 2 g(x, y) = A( (1 _ {3x}(1 _ {3y) + 21 _ {3x g(x, y) + g(x, y) ), (1.9)

where A = {3A if {3 > 1, and A = A otherwise. Obviously, 9 majorizes I up to order 2. Furthermore, by iteration of equation (1.9), 9 has the property that the coefficient of yk consists of a sum of terms of the form a/(I- {3x)' with a E 1R+, lEN, l ::; k. Assume that I is majorized by 9 up to order k in y. As

yk {3y yk yT (1 _ {3x)1 - k 1 - {3x (1 - {3x)1

yT I(x, y) is majorized up to order (k+ 1) by {3yg(x, y)/(I- {3x), and hence, by equations (1.8), (1.9), I is majorized by 9 up to order (k + 1), so I is majorized by g.

The solutions to the quadratic equation (1.9) are

g(x,y) = -! (2-y- - :)±J! (2-y- _ :)2 2 1 - {3x A 4 1 - {3x A (1 - {3x}(1 - {3y).

Both solutions are obviously analytic functions in (x, y), as the argument of the square root is an analytic function with nonvanishing zeroth degree term, the solution with the minus sign being the desired solution, which starts with a term quadratic in y. Hence I is majorized by a convergent power series, and 2 is shown. Q.E.D.

Since the Ehresmann connection defined by X is fiat, the horizontal distribution is integrable. Due to the 6-term in (1.4), this distribution is transversal to the zero section Z, and for a sufficiently small tubular neighborhood U C T P of Z each leaf in U intersects Z in exactly one point. Hence, we can define an "exponential mapping" EXP p : TpP n U ~ P for any pEP which maps vp to the unique intersection point of the leaf through Vp and the zero section in TP (Figure 2).

230 C. Emmrich and A. Weinstein

Figure 2. The "exponential mapping" for a flat nonlinear connection is found by following parallel sections from the fibres to the zero section.

With this definition, the normalization condition (1.6) with a = 0 is distinguished by the following theorem:

Theorem 1.3 The mappings EXPp and the usual exponential mappings expp coincide on Un TpP for all pEP iff the choice 8-1(X) = 0 is made in {1.6}.

Proof. To prove the theorem, we first observe that X determines a nat­ural notion of autoparallel curves: Any curve 'Y in P may be lifted to a curve in TP by taking the derivative 'Y. A curve 'Y is called autoparallel iff 'Y is horizontal, i.e., tangent to the horizontal distribution defining the connection.

In local coordinates, this condition is of the form:

(1.10)

where X~(x,Y)l,dxk is the local coordinate expression for X. The connectfon between autoparallel curves and the 'exponential map­

ping' EXP is particular simple in the case of P = ]Rn with the standard flat connection 8 (Figure 1). In this case, the equation for autoparallel curves is just xi(t) = _±i(t) with the solution

±(t) = ±(O)e-t , x(t) = x(O) + ±(0)(1- e-t ).

The term _±i(t) coming from the 8-term in X tot can be interpreted as a friction term slowing the particle down as it approaches the zero section (Figure 1). In particular, EX P(x(O), v(O» = limt-+cx> x(t) = x(O) + v(O), and EX P coincides with the usual exponential mapping of the connection 8. In the general case, the interpretation of the first term on the right hand side of (1.10) as a friction term is still valid.

The Differential Geometry of Fedosov's Quantization 231

Coming back to the general case, note that the last term in (1.10) is precisely 6-1(X). Suppose that this is zero. Then a reparametrization t ~ -log(l - r) yields the usual geodesic equation corresponding to the connection 8 for x(r) ~ x( -log(l-r)). In particular, for sufficiently small initial velocity v, x(r) is defined for 0 :$ r :$ 1, and x(t) is defined for 0:$ t < 00.

As limt-+oo r(t) = limt-+oo(1 - e-t ) = 1, we get: limt-+oo x(t) = lim-r-+l x(r) = exp(v) with v = x(O). On the other hand, by construc­tion the curve (x, x) (t) in T P is contained in the leaf of the horizontal foliation through v. As limt-+oo x(t) = limt->oo e-t lTx(r) = 0, limt-+oo x(t) is just the intersection point of the leaf through v and the zero section, Le. EXP(v). Hence, EXP(v) = exp(v) and we have proved that the condition 6-1(X) = 0 is sufficient for the maps EXP and exp to coincide.

To prove the converse we assume EXP p = expp for all PEP, and 6-1 X arbitrary. We first show the following lemma:

Lemma 1.4 Let'Y be a geodesic corresponding to the connection 8, and let i' : lit -- P be the reparametrized geodesic: t 1-+ 'Y(1 - e-t), i.e., a solution to the differential equation

··i (t) - . i (t) ri ( ). k ·1 X - -x - kl X X X • (1.11)

Then, for all t:

exp.:y(t) (.:y(t») = exp.:y(O) (.:y(0») = t~~ i'(t)

Remark: The lemma says that a solution of (1.11) is a reparametrized geodesic that is slowed down by the 'friction term' -x just to that extent that exp(.:y(t)) is constant.

Proof of the lemma: Denote by 'Y2 the geodesic starting at i'(to) with initial velocity ..y(to)

for some fixed time to. As follows from the first part of the proof of the­orem 1.3 above, i'2(t) = 'Y2(1 - e-t ) defines again a solution of (1.11). Obviously, this solution is obtained from i' by a constant shift in time, as .:y2(0) = ..y(to). Hence, limt->oo i'(t) = limt-+oo i'2(t). On the other hand, we have already proved that limHoo i'(t) = exp.'Y(O) (..y(0)) (and hence

limt->ooi'2(t) = exp.:y2(O) (..y2 (0))) , so the lemma follows. Q.E.D.

Using this lemma we will finish the proof of Theorem 1.3, i.e., we will show that the condition 8-1 X = 0 is a necessary condition for the maps EXP and exp to coincide.

Assume that the curve t 1-+ 1n(1 - e-t ) in T P is not completely contained in the leaf through Vq E TP, where 'Y : t 1-+ 'Y(t) is a geodesic

232 C. Emmrich and A. Weinstein

of a with 1'(0) = vq • By continuity there is a open subset 0 C R such that w(t) := 10(1 - e- t ) is not contained in the leaf through Vq for any tEO. By assumption and the lemma, EXP(w(t)) = exp(w(t» = exp(vq ) = EXP (vq ), so we get an infinity of different leaves intersecting the zero section in the same point, in contradiction to the transversality of the horizontal distribution.

Hence, the curve t 1-+ *,1'(1 - e-t ) is completely contained in the leaf through vq , i.e., it is a horizontal curve, and therefore must fulfill both equations (1.10) and (1.11). Obviously, this is only possible if the difference between the two equations vanishes, i.e.,

and Theorem 1.3 is proved. Q.E.D.

Remark: Even when 6-1 X i= 0, the equality EXP(v) = limt----+oo x(t), where x(t) is the solution of (1.10) with X(O) = v, still holds. To prove this statement we show that the 'friction term' in (1.10) insures that limt----+oo x(t) = 0 for sufficiently small initial velocity v. However, this is easy to see: Multiplying (1.10) by Xi and summing over i, we get:

As X has degree at least two in y, for sufficiently small velocity

I Lxi (r~,(X)xkx' + Xk(x,x)xk) 1< f Lxixi i i

for some f with 0 < f < 1. Hence,

e-2(1+e)t L Xi(O)xi(O) < L Xi (t)xi(t) < e-2(1-e)t L Xi(O)Xi(O), iii

and lim x(t) = 0

t----+oo

but the curve (x(t),x(t)) does not intersect the zero section at a finite time.

The Differential Geometry of Fedosov's Quantization 233

7. The formal exponential map

In the last section we have shown how to construct the mapping EXP in the analytic case, where the formal flat connection was convergent and defined a flat Ehresmann connection in the usual sense on some neighborhood of the zero section. In the non-analytic case we cannot expect the series for X to be convergent. However, we will show that we can define a formal exponential map, i.e., an co-jet of mappings from the fibre TpP to P, and we will show that for this map Theorem 1.3 still holds, if we replace expp

by its co-jet as well. In order to define the formal exponential map we first show the following

lemma for real-analytic manifolds:

Lemma 1.5 Let P be a real-analytic manifold, 8 an analytic connection and a an analytic function, X the solution of (1.5) with the normalization condition (1.6). Then the k-jet of EXPp at Op E TpP for arbitrary k E N, pEP is completely determined by the k-jet at p of the connection form corresponding to 8 and the (k + I)-jet at Op of a.

Proof. It is sufficient to consider some small neighborhoods V C P of pEP and TV C T P of Op E TpP. We choose coordinates (Xi) on V and show first that the k-jet of EXP p is determined by the k-jet of X.

By the definition of the map EXP p' we may compute the components EXP~ of EXP p in the chosen coordinate system by taking the covariant constant continuation fi to TV C T P of the coordinate functions xi and restricting them to the fibre TpP:

EXP~(y) = t(y) I Tl'P

with

where Z denotes the zero section in TP. As the constructed connection is flat, we know that such functions Ii

exist. Rather than constructing them by the iterative procedure of [10), we will use a more geometric approach. The condition D Ii = 0 reads locally:

8 fi Mm 8 fi 8xk + k 8ym = 0, (1.12)

where M is the matrix with elements

(1.13)

where 15k' equals 1 if m = k, and 0 otherwise. Due to the 15k' coming from the solder form the matrix valued function M is invertible, and the inverse

234 C. Emmrich and A. Weinstein

M- 1 is analytic at Op again. Furthermore, its k-jet at Op only depends on the k-jet of Mat Op.

Hence, we can solve (1.12) for ~ Ji yielding

~ fi + (M-1)m~ fi = 0, 8yk k 8xm

(1.14)

where the zero-jet of M-1 at Op is just the unit matrix. fi is given for y = 0, and we know that this set of partial differential equations has a unique solution, since the connection is flat. To compute it, we can integrate the differential equations successively: The equation for I!r fi determines fi

on the y1-axis, the equations for a~k fi with 1 ~ k ~ m determine fi on the subspace spanned by the first m basis vectors. In each step the solution is an analytic function on the respective subspace, depending analytically on the initial conditions, by the Cauchy-Kovalevskaya theorem. Hence, fi is analytic at Op. Furthermore, by comparing coefficients, we see that the k-jet of fi only depends on the k-jet of M- 1 , and hence of M.

Since the operator 6-1 contains a factor yi, it follows from the iteration formula (1.7) for X that the k-jet of X is determined by the k-jets of the Christoffel-symbols r;k of 8 and the (k + I)-jet of a. By the definition (1.13) this is true for M as well, and the lemma is proved. Q.E.D.

Using this lemma, we can define a formal exponential map in the non­analytic case: We use the topology on the oo-jets generated by the basis of open sets consisting of sets of oo-jets whose k-jets agree for some kEN, i.e., a sequence (/) of oo-jets converges to an oo-jet j, if for any kEN there is an N(k) EN auch that the k-jets of jr and j agree for r > N(k).

In this topology the set of oo-jets of analytical functions forms a dense subset of the set of all oo-jets. (The polynomials are already dense.) Hence, we can define the oo-jet g'(EXPp) of EXP at p in the non-analytic case by the continous continuation to all oo-jets of the map that assigns to the pair (8,j;'(a)), consisting of an analytic connection and the oo-jet of an analytic function a defining the normalization condition for X, the oo-jet j;'(EXPp)' This continuation exists and is unique by Lemma 1.5.

Theorem 1.6 Theorem 1.3 still holds in the non-analytic case, if we re­place EXPp , and expp by their oo-jets.

Proof. If we impose the normalization condition (1.6) with a = 0 the direct statement of the theorem is an immediate consequence of Theorem 1.3, Lemma 1.5, and the fact that the oo-jets of analytic functions are dense in the chosen topology.

To show the converse, we use the fact that j;' (expp ) coincides with j;'(EXPp) for the choice a = 0 in (1.6) by the first part of the theorem.

The Differential Geometry of Fedosov's Quantization 235

We denote for the moment the corresponding X by XO. Now, assume that the k-jet of a in (1.6) is different from zero for some k > 2. Then, by (1.7), the (k - I)-jet of X and XO are different as well. This implies, that the (k - I)-jets of M and M-1 in (1.12), (1.14) are different from those for a = 0, so the (k -I)-jets of the corresponding jets i;'(EXPp ) are different. Q.E.D.

8. Flat symplectic connections

So far, we have studied the problem of constructing a flat connection for an arbitrary manifold P, without any additional structure. However, if we are interested in getting closer to the quantum case, then P should be a classical phase space with its symplectic structure, and :F(P) its Poisson algebra.

In this case, the Weyl-algebra bundle of [10] is replaced by A = :F(TP) (or A = r(uPEP .1000 (TpP, R))) with the fibrewise Poisson-structure {, hib' The flat connection must be compatible with this additional structure, Le., the following equation should hold for all f, 9 E A:

D{f,ghib = {Df,ghib + {f,Dg}fib. (1.15)

Hence, {) has to be a symplectic connection and X (v) has to be a symplectic vector field on TxP for any v E TxP. As the fibres are vector spaces, X must be globally hamiltonian, Le., there must be an A-valued one-form r such that the components of the one form r are minus the hamiltonian functions of the components of the vector-valued one-form X. (The minus sign is chosen in order to guarantee that the Lie bracket of X with itself corresponds to the Poisson bracket of r with itself.)

In this case, the normalization condition 15-1 X = 0 is in general not admissible, as it will not give a symplectic vector field. This statement is an immediate consequence of the fact that the usual exponential mapping of a symplectic connection is generally not a local symplectomorphism, and the following theorem:

Theorem 1.7 The mapping EXP(p) is a local symplectomorphism iff XCv) is a hamiltonian vector field for any vET P.

Proof. Let XCv) be a hamiltonian vector field for any v E TP, with X = Xr for some A-valued one-form r. Then D is of the form:

and has the property (1.15).

236 c. Emmrich and A. Weinstein

We have to show:

(1.16)

for all j, 9 E :F(P) , where {,}p is the Poisson bracket on the fibre TpP induced by the constant symplectic form w(p), i.e., the restriction of {, }fib to the fibre TpP.

For any a E :F(P) let a(a) denote the unique covariant constant con-tinuation of a, i.e., a(a) E A, Da(a) = 0, a(a) I = a, where we have

PCTP identified P with the zero section in TP.

By construction of EXP we have:

(1.17)

Obviously, due to the o-term in X tot , a(a) is locally of the form:

() . 2 a(a)(x, y) = a(x) + -() . a(x)y' + O(y )

x'

and hence:

{a(f),a(g)}fibl p = {J,g}, D{a(j),a(g)}fib = 0 (by (1.15»

=} {a(f),a(g)}fib = a({J,g})

for any j, 9 E :F(P) by uniqueness of the covariant constant continuation of a function in':F(P). Using (1.17), equation (1.16) follows. Q.E.D.

For a given linear symplectic connection the set of its nonlinear "flat­tenings" satisfying (1.15) is in one-to-one correspondence with the functions j E A at least of degree 4 in the fibres by the condition: o-lr = j, where r is the unique one form in A 0 A(P) which is a hamiltonian for X and vanishes on P C TP. Indeed, as any X is at least of second degree in the vertical coordinates, r is at least of third, and o-lr at least offourth degree.

On the other hand, given j, we miW write the analog of equation (1.7) for r instead of X, which is essentially the same (up to signs, since X is minus the hamiltonian vector field of r, and the replacement of the commutator of vector fields by the Poisson bracket of functions.) Hence, given j, we can uniquely construct r as the solution of the equation:

0= R - Or + ar + 1/2{r,r}. (1.18)

with the condition (1.19)

The Differential Geometry of Fedosov's Quantization 237

where the A-valued two-form R is the hamiltonian for the curvature R considered as a two form on P with values in the vertical vector fields on TP, locally given by R = (1/4)wijR{lmyiykdxkdxm. This solution may again be iteratively computed:

r = 8f + 8-1(R + ar + 1/2{r,r}). (1.20)

We observe that the connection on U c T P can be easily recon­structed from the mapping EXP: The leaf through any Op E T P is given by UqEP EXp;l( {Op}), and the connection is defined by the tangent distri­bution to those leaves. Since the mappings EXP p for different symplectic connections a and different 'normalization conditions' (1.19) are all local symplectomorphisms, this observation implies the following theorem:

Theorem 1.8 The different Ehresmann connections constructed from dif­ferent symplectic connections and different choices of f are related by fibre­wise symplectomorphism, i.e., by a section in the bundle UPEP Symop(TpP) where Sym Op (TpP) denotes the group of (infinite jets oJ) symplectomor­phisms of TpP with fixed point Op E TpP.

This statement is the symplectic analog of Theorem 4.3 in [10] for the quantum case, which states that two different 'abelian connections' D, jj are related by some inner automorphism U such that jj = D- [DU oU-I, .]. (The theorem is formulated only for the normalization condition 8-1r = O. However, this condition is not used in the proof, so it is valid for the more general normalization conditions studied by us as well.)

For the condition 8-1r = 0, EXP is in a suitable sense a 'minimal defor­mation' of the usual exponential mapping such that the resulting mapping is symplectic: namely, as the symplectic connection a defines a splitting of TT P in horizontal and vertical subspaces, we can define the accelaration a(vp ) of autoparallel curves due to X and the friction term -8 at a point vp E T P as the difference of the tangent vectors to the lifted autoparallel curve and the lifted geodesic corresponding to a through that point, which is always a vertical vector. We identify T(TpP) and TpP, and denote by j the vertical lift: j: TpP -+ T(TpP), v t-t vl . In order to avoid notational confusion, we denote by wp the symplectic form on the vector space TpP induced by the symplectic form w on P.

Theorem 1.9 The condition 8-1r = 0 for a solution of (1.18) is equiva­lent to the requirement that the acceleration a(vp) is contained in the wp­complement ofv~ for all p E P,vp E TpP, i.e, wp(v~,a(vp» = 0 Vvp E TP.

Proof. Any solution r of (1.18) is of the form (1.20) for some f. a( vp ) may be written as

238 c. Emmrich and A. Weinstein

where the first term comes from the friction term -6 and aX(vp) is the acceleration due to X. For arbitrary f, we conclude from (1.10) that aX (vp )

is just (6-1 X)(vp ), which is nonzero even if 6-1r = 0, as 6-1 explicitly contains the vertical coordinates y.

Using the definition of 6-1 we get:

6-1X = I; 1 Xo-l r + (w#(I-l r»l, (1.21)

where X f denotes the hamiltonian vector field on the fibres of T P cor­responding to f (Le., w#(dyf), and l.Jl, I-I are linear operators acting on monomials in y of total degree i by multiplication with i¥ and i-I, respectively.

By (1.20) r is of the form r = 6f + 6-1a for a two form a E A ® A(P). Setting Vp = ~ yi a~. and using (1.21) we get:

W(/; 1 Xj, Vp1) + w(w(p)#(I-l(6f + 6- 1a»1, Vp1)

(6f)(Vp) + (~6-1a)(Vp) (6f)(Vp),

where the last equality holds because for each monomial in y, the expres­sion (6- 1a)(Vp) is - up to a real factor - just a(Vp, Vp) = o. Hence, wp(vJ,a(vp» = 0 VVp E TP is fulfilled for f == o.

On the other hand, since (6 f) (vp ) = dd>. f (x, AY) I >.= l' it follows from the fact that

wp(vJ,a(vp» = 0 Vvp E TP

that f(x, y) has to be independent of y and hence zero, as the constant term in f must vanish. Q.E.D.

References

[1] Bayen, F., Flato, M., Fronsdal, C., Lichnerowicz, A., and Sternheimer, D., Deformation theory and quantization, I and II, Ann. Phys. 111, (1977), 61-151.

[2] Berezin, F.A., Quantization, Math USSR Izv. 8 (1974), 1109-1165.

[3] Bohm, D., Quantum Theory, Prentice-Hall, New York, 1951.

[4] Boutet de Monvel, L. and Guillemin, V., The spectral theory of Toeplitz operators, Annals. of Math. Studies 99, Princeton University Press, Princeton, 1981.

The Differential Geometry of Fedosov's Quantization 239

[5} De Wilde, M. and Lecomte, P., Existence of star-products and of for­mal deformations of the Poisson Lie algebra of arbitrary symplectic manifolds, Lett. Math. Phys. 7 (i983), 487-496.

[6} Donin, J., On the quantization of Poisson brackets, Advances in Math. (to appear).

[7} Fedosov, B., Formal quantization, Some Topics of Modem Mathemat­ics and their Applications to Problems of Mathematical Physics (in Russian), Moscow (1985), 129-136.

[8} Fedosov, B., Index theorem in the algebra of quantum observables, Sov. Phys. Dokl. 34 (1989), 318-321.

[9} Fedosov, B.V., Reduction and eigenstates in deformation quantization, Advances in Partial Differential Equations, Akademie Verlag, Berlin (to appear).

[1O} Fedosov, B., A simple geometrical construction of deformation quan­tization, J. Diff. Geom. 30 (1994), 327-333.

[11] Flato, M., and Sternheimer, D., Closedness of star products and coho­mologies, this volume, pp. 241-259.

[12] Kobayashi, S. and Nomizu, K., Foundations of differential geometry, Interscience, New York, 1963.

[13] Krantz, S.G. and Parks, H.R., A Primer of Real Algebraic Functions, Basler Lehrbiicher vol. 4, Birkhauser, Basel, 1992.

[14] Nest, R. and Tsygan, B., Algebraic index theorem for families, Ad­vances in Math. (to appear).

[15] Omori, H., Maeda, Y., and Yoshioka, A., Weyl manifolds and defor­mation quantization, Advances in Math. 85 (1991),224-255.

[16] Weinstein, A., Symplectic manifolds and their lagrangian submani­folds, Advances in Math. 6 (1971),329-346.

[17] Weinstein, A., Classical theta functions and quantum tori, Publ. RIMS Kyoto Univ. 30 (1994), 327-333.

[18] Weinstein, A., Deformation quantization, Seminaire Bourbaki, 46eme annea, 1993-94, nO 789, juin 1994 (to appear in Asterisque).

Department of Mathematics, University of California, Berkeley, CA 94720, USA

Received November 18, 1993

Closedness of Star Products and Cohomologies

Moshe Flato and Daniel Sternheimer

Dedicated to Bert Kostant with friendship and appreciation

Abstract

We first review the introduction of star products in connection with deformations

of Poisson brackets and the various cohomologies that are related to them. Then

we concentrate on what we have called "closed star products" and their relations

with cyclic cohomology and index theorems. Finally we shall explain how quan­

tum groups, especially in their recent topological form, are in essence examples

of star products.

1. Introduction: quantization

1.1 Geometry. The setting of classical mechanics in phase-space has long been a source of inspiration for mathematicians. But (according to writing on a wall of the UCLA mathematics department building) Goethe once said that "Mathematicians are like Frenchmen: they translate everything into their own language and henceforth it is something completely different". Being French mathematicians, we shall give here a flagrant illustration of that sentence, though not going as far as Bert Kostant's cofounder of geometric quantization (Jean-Marie Souriau) who derived symplectic for­malism from the basic principles that are the core of the French "mecanique rationnelle" established by Lagrange.

The symplectic formalism is obvious in the Hamiltonian formulation on flat phase-space n U , and prompted in the fifties French mathematicians like Paulette Libermann and Georges Reeb to systematize the notion of a symplectic manifold. Parallel to these developments came the introduction of quantum mechanics (first called "mecanique ondulatoire" in France un­der de Broglie's influence). Then (which brings us close to our subject here) Dirac [1] introduced (both in the classical and in the quantum domain) his notion of "constrained mechanics", when external constraints restrict the degrees of freedom of phase-space. For mathematicians this is nothing but restricting phase-space to a submanifold of some n2l endowed with a Poisson manifold structure [2] (second class constraints give a symplectic submanifold); this restriction permits a nice and compact formulation of

242 M. Flato and D. Sternheimer

classical mechanics, but was of little help in the quantum case where people needed some reference to the canonical formalism on flat phase-space, as exemplified by the Weyl quantization procedure [3].

Then quite naturally Kostant (coming from representation theory of Lie groups) and Souriau (coming from the symplectic formulation of clas­sical mechanics) introduced independently [4] what is now called geometric quantization. The idea is to somehow select, via a polarization, a La­grangian submanifold X of half the dimension of the symplectic manifold W so that locally W will look like T* X, and quantization can be done on L2(X); and in the meantime to work at the prequantization level on L2(W). That idea, quite efficient for many group representations, ran into serious problems (now well-known) on the physical side; in particular few observables were "quantizable" in that sense.

1.2 Deformations. The idea that the passage from one level of physical theory to a more evolved one is done through the mathematical notion of deformation became obvious only recently [5], but many people certainly felt that something of this kind must occur. On the space-time invariance level (Galileo to Poincare to De Sitter) it is simple to formulate because all objects are Lie groups. When interactions (nonlinearities) occur, it is more intricate (and requires the systematization of the notion of nonlin­ear representations [6]). For quantum theories, in spite of hints in several expressions (like "classical limit"), the "quantum jump" from functions to operators remained. Our approach, that started about 20 years ago [7-11], and is now often referred to as "deformation quantization", showed that there is an alternative (a priori more general) and autonomous formulation in terms of "star" products and brackets, deformations (in Gerstenhaber's sense [12]) of the algebras of classical observables (Berezin [13] has inde­pendently written a parallel formulation in the complex domain, but not in terms of deformations).

The importance of algebras of observables (especially C* algebras) in quantum theories has even spilled into geometry with the non-commutative geometry of Alain Connes [14] and its developments around algebraic in­dex theory (generalizing the Atiyah-Singer index theorems for pseudo­differential operators), and with the exponential development of quantum groups [15]. In this paper, after a survey of the origins (Section 2), we shall (in Section 3) indicate that what we have called "closed star products" [16] permits a parallel treatment of the former, and (in Section 4) that the latter, once formulated in a proper topological vector space context, are essentially examples of star products. In each case there are appropriate cohomologies to consider, e.g., cyclic for closed star products and bialgebra cohomologies for quantum groups, that are more specific than the tradi-

Closedness of Star Products and Cohomologies 243

tional Hochschild (and Chevalley) cohomologies. An alternative name for our approach could thus be "cohomological quantization". It stresses the importance of cohomology classes in all our approach and leads naturally to ideas like "cohomological renormalization" in field theory (when phase­space is infinite-dimensional) where "more finite" cocycles can be obtained [17] by subtracting "infinite" coboundaries from cocycles to define star products equivalent to (but different from) that of the normal ordering.

2. Star products and cohomologies

2.1 Deformations of Poisson Brackets. Let W be a symplectic man­ifold of dimension 2£, with (closed) symplectic 2-form Wj denote by A the 2-tensor dual to W (the inner product by -w defines an isomorphism J.t between TW and T* W that extends to tensors). The Poisson bracket can be written as P(u, v) = i(A)(du 1\ dv) for u, v E N = Coo(W). Some of the results described here are valid when W is a Poisson manifold (where A is given, with Schouten autobracket [A, A] = 0 - the analogue of closedness for w - but not necessarily everywhere nonzero)j the dimension need not be even and a number of results hold when the dimension is infinite, which is the case for field theory (Segal and Kostant [18] were among the first to consider seriously infinite-dimensional symplectic structures). We shall however not enter here into the specifics of these questions.

a. A deformation of the Lie algebra (N, P) is defined [12] by a formal series in a parameter v E C:

00

[u,v]", = P(u,v) + L>rar(u, v), for u, v E N (or N[[v]]) (1) r=l

such that the new bracket satisfies the Jacobi identity, where the ar are 2-cochains (linear maps from N 1\ N to N). In particular a1 must be a 2-cocycle for the Chevalley cohomology H* (N, N) (for simplicity we shall write H*(N), and similarly for Hochschild cohomology) of (N, P). As usual, equivalences of deformations are classified at each step by H2 (N) and the obstructions to extend a deformation from one step (in powers of v) to the next are given by H3(N) [11,12]. Whenever needed we shall put on the formal series space N[[vll its natural v-adic topology.

Moreover, it can be shown that one gets consistent theories by restrict­ing to differentiable (resp. I-differentiable) cochains and cohomologies, when the ar are restricted to being bidifferential operators (resp. bidiffer­ential operators of order at most (1,1». This has the advantage of giving finite-dimensional cohomologies with simple geometrical interpretation. In

244 M. Flato and D. Stem.heimer

particular one has

(2)

if w is exact (it can be smaller if not). One can also restrict to differentiable cochains that are null on con­

stants (n.c. in short), i.e., Cr(u,v) = 0 whenever either u or v is constant, and again get a consistent theory. In that case iI?diff,nc(N) = H 2 (W,JR.), the de Rham cohomology. Similar results hold for iI~iff(N), the obstruc­tions space [19]; in particular, iI~iff nc(N) is isomorphic to Hl(W)E9H3(W) (for w exact, modulo some conditio~ on a 4-form; without it and/or without the n.c. condition, the space may be slightly larger). This explains that, when the third Betti number of W, b3 = dim H3(W), vanishes, J. Vey was able to trace the obstructions inductively into the zero-class of iI3 (N) and show the existence of such deformed brackets (the condition b3 = 0 is not necessary, as follows from the general existence theorems for star products that we quote later). Replacing in all the above "differentiable" by "local" gives essentially the same results for the cohomology [19].

b. In the differentiable n. c. case one thus gets that, if b2 = dimH2(W) = 0, there is (modulo equivalence) only one choice at each step, coming from the Chevalley cohomology class of the very special co­cycle S~ given, on any canonical chart U of W, by

where C(Xu) is the Lie derivative in the direction of the Hamiltonian vector field Xu = J-L-l(du) defined by u E Nand r is any symplectic connection (rijk totally skew-symmetric; i,j, k = 1, ... ,2£) on W. The Chevalley cohomology class of Sf is independent of the choice of r. On JR.2l (with the trivial flat connection) it coincides with p3, when we denote by pr the rth

power of the bidifferential operator P. It is [9-11] the pilot term for the Moyal bracket [21] M, given by (1) where (2r+l)!Cr = p2r+1, i.e., the sinh function of P (the only [11] function of P giving a Lie algebra deformation). In the Weyl quantization procedure, M corresponds to the commutator of operators (when we take for deformation parameter v = ~ili).

2.2 Deformations of associative algebras. On N (or N[[v]]) we can consider the associative algebra defined by the usual (pointwise) product of functions. Its deformations are governed by the Hochschild cohomology H*(N), and here also it makes sense to restrict to local or differentiable (n. c. or not) co chains and cohomologies. All the latter cohomologies are in

Closedness of Star Products and Cohomologies 245

fact the same: HP(N) = I\P(W), the contravariant totally skew-symmetric p-tensors on W; if b denotes the Hochschild coboundary operator, any (local, etc.) p-cocycle C is of the form C = D + bE with D E I\P(W) and E a (local, etc.) (p - I) cochain. This result was obtained in an algebraic context in [20].

a. In order to relate to the preceding theory and thereby reduce the (a priori huge) possibilities of choices, and also of course because this is the physically interesting case, we shall be interested only in deformations such that the corresponding commutator starts with the Poisson bracket P, what we call "star products":

C1(u,v) - C1(v,u) = 2P(u,v) (5)

where the cochains Cr are bilinear maps from N x N to N. In the local (etc.) case, one necessarily has C1 = P + bTl, and therefore any (local, etc.) star product is equivalent, via an equivalence operator T = I + vTl. T1 a differential operator, to a star product starting with C1 = P. Note that, in the differentiable case, an equivalence operator T = I + L~lvrTr between two star products is necessarily [11] given by a formal series of differential operators Tr (n. c. in the n. c. case). Star products are always nontrivial deformations of the associative algebra N because P is a nontrivial 2-cocycle for the Hochschild cohomology (a coboundary can never be a bidifferential operator of order (I, I ».

The case when the cochains Cr are differentiable and odd or even, to­gether with r (what we call the parity condition, Cr(u,v) = (-ltCr(v,u» is simpler and we considered it first [11]. However the parity condition is not always needed, and the differentiability condition is sometimes not com­pletely satisfied. This is especially the case when one deals with what we call "star representations" of (semi-simple) Lie groups G, by star products on coadjoint orbits. There, the orbits being given by polynomial equations in the vector space of the dual g* of the Lie algebra g of G, the Cr will in general be bi-pseudodifferential operators on the orbits. It would thus be of interest to introduce another category of star products, when the co chains are algebraic functions of bidifferential operators; the related cohomologies would probably not be very different from the differentiable case ones. On the other hand, restricting to I-differentiable cochains is not of much inter­est here (by opposition to the Lie algebra case [2]) since one then loses all connection to quantum theories because the cochain Sf is lost (the order of differentiation is either I or unbounded).

246 M. Flato and D. Sternheimer

h. From (3) one gets a deformed bracket by taking the commutator 2~(u*v-v*u), which gives a Lie algebra deformation (1) with 2(\_1(U, v) = Cr(u,v) - Cr(v,u). In contradistinction with the Lie case, the Hochschild cohomology spaces are always huge but the choices for star products will be much more limited because of the associated Lie algebra deformations.

In particular when the Cr are differentiable and satisfy the parity con­dition, the C2r+l being n. c. (what is called a "weak star product"), any star product is equivalent [22] to a "Vey star product", one for which the r!Cr have the same principal symbol as Pf, the rth power of the Poisson bracket expressed with covariant derivatives " relative to some symplec­tic connection r, i.e., on a local chart U (with summation convention on repeated indices):

nr(u v)1 - Aidl Airjr". . u"· . v £r , u - ... v.1 ...• r VJ1·.·Jr , u,vEN. (6)

For such products the most general form of the first terms are

(7)

where T(2) is a differential operator of order at most 2, b denotes the Cheval­ley coboundary operator and A2 is a 2-tensor, image (under p-l) of a closed 2-form [19, 22]. A somewhat general expression can also be given for C4 [19], but it is much more complicated. For higher terms, no explicit formula was published (Jacques Vey knew more or less how to do it for C5 ) but there exists an algorithmic construction due to Fedosov [23] in terms of a symplectic connection that gives a class of examples term by term.

What happens here (assuming the parity condition) is that the Lie al­gebra (Le., the odd co chains) determines inductively the star product from which it originates; the only freedom is the possible addition, at each even level, of multiples of uv to the cochains (and to the equivalence operators). The parity condition is of course satisfied at level 0 and can (by equiva­lence) be assumed at level 1 for star products, but the Lie algebra will in general (except when b2 = 0) give enough information only on the odd part of the co chains Cr.

2.3 Existence, uniqueness and examples of star products.

a. Existence. On]R2l with the flat symplectic connection one has the Moyal star product and bracket:

U *M V = exp(vP)(u, v) M(u, v) = v-1sinh(vP)(u, v). (8)

The idea is to take such star products MOl on Darboux charts UOI for any symplectic Wand glue them together. This cannot be done brutally (when

Closedness of Star Products and Cohomologies 247

b3 # 0 the topology of the manifold hits back). But N[[vll can be viewed as a space of flat sections in the bundle of formal Weyl algebras on the tangent spaces of W (a Weyl algebra is generated by the canonical com­mutation relations [Xi, xj] = 2vAij I); a flat connection on that bundle is algorithmically constructed [23] starting from any symplectic connection on W. Pulling back the multiplication of sections gives a star product [24], which can also be seen as obtained by the juxtapositions of star products TaMa on each Ua, when the equivalence operators Ta are such that all TaMa and Tf3Mf3 coincide on Ua n Uf3 (the Darboux covering is chosen locally finite). These star products can be taken to be differentiable n. c. (d. n. c. in short) and satisfying the parity condition.

Earlier proofs of existence were done first in the case b3 = 0, then for W = T* X with X parallelizable, and shortly afterwards, for any symplectic (or regular Poisson) manifold; but that proof was essentially algebraic, while we now better see the underlying geometry.

b. Uniqueness. Formula (7) is very instructive about what happens. Indeed whenever we have two Lie algebra deformations equivalent to some order, after making them identical to that order by an equivalence, the difference of the cochains is a cocycle of the form given by C3 in (7), in the d.n.c. case of course. Therefore we have at each step (for the bracket) at most 1 + b2 choices modulo equivalence, and we see exactly where the second de Rham cohomology enters: the "I" stands for the Moyal bracket, and the b2-dimensional space comes from what we called in [7] "inessential" I-differentiable deformations that are obtained by deformations of the 2-tensor A, i.e., by deformations of the closed 2-form W (adding an exact 2-form gives an equivalent deformation).

For star products (P being a nontrivial Hochschild cocycle), the "start­ing point" becomes the Moyal product, and the equivalence classes are clas­sified by the second de Rham cohomology. Indeed we know now (cf. [23, 24]) that it is always possible to avoid the obstructions; and if two star products are equivalent to order k, once they are made to coincide at that order, the skew-symmetric part of their difference at order k + 1 determines a closed 2-form that is exact iff they are equivalent to order k + 1. (This follows from an argument due to S. Gutt, similar to those of [19, 22]).

In particular, also in the d. n. c. case and without the parity condition, when ~ = 0, the Moyal- Vey product is unique. In that case one can choose a star product satisfying the parity condition (denote its cochains by C~); any other d. n. c. star product (with cochains Cr ) can then step by step be made equal to the chosen one by the above-mentioned argument: at the first step where Ck - C~ is nonzero, it is of the form Dk + bEk with Dk a closed, thus exact, 2-form: Dk = dFk (here Dk(U,V) is defined as

248 M. Flato and D. Sternheimer

Dk(Xu ® Xv), and similarly, Fk(U) == Fk(Xu)); the equivalence will then be extended to the next order by 1- vk- 1 Fk - vk Ek.

c. Examples. The various orderings considered in physics are the inverse image of the product (or commutator) of operators in L2(JRl ) under the Weyl mappings

where it is the inverse Fourier transform of u, P and Q are operators satisfying the canonical commutation relations [Pa , Qa] = iMa/1(a, f3 = 1, ... ,l), w is a weight function, 2v = in and the integral is taken in the weak operator topology. Normal ordering corresponds to the weight w(e,.,,) = exp( -He ± .,,2», standard ordering (the case of the usual pseu­dodifferential operators in mathematics) to w(e,.,,) = exp( -~e.,,) and Weyl (symmetric) ordering to w = 1. Only the latter is such that C1 = P (e.g., standard ordering starts with the first half of P) but they are all mathe­matically equivalent via the Fourier transform of w. (Physically they give different spectra, when we define the star spectrum as indicated here be­low for the image of most classical observables; in fact two isospectral star products are identical [25]).

Other examples can be obtained from these products by various devices. For instance one can restrict to an open submanifold (like T*(JRl - {O}», quotient it under the action of a group of symplectomor­phisms and restrict to invariant functions; one can also transform by equiv­alences, or look (cf. below) for G-invariant star products. A variety of physical systems can thus be treated in an autonomous manner [II]. The simplest of course is the harmonic oscillator, which relates marvelously to the metaplectic group (dear to Bert Kostant). But other systems, such as the hydrogen atom, have also been treated from the beginning - which is not the case of geometric quantization.

An essential ingredient in physical applications is an autonomous spec­tral theory, with the spectrum defined as the support of the Fourier-Stieltjes transform of the star exponential Exp(tH) = E:'=o ?r(tH/in)*n, where the exponent *n means the nth star power. Such a spectrum can even be defined in cases when operatorial quantization would give nonspectrable operators (e.g., symmetric with different deficiency indices). The notion of trace is also important here, and will bring us to closed star products. Interestingly enough, the trace of the star exponential of the harmonic oscillator was already obtained in 1960 by Julian Schwinger [26] (within conventional theory, of course).

Closedness of Star Products and Cohomologies 249

d. Groups. ]R2l is the generic coadjoint orbit of the Heisenberg group in 1ft = ]R2l+1; the uniqueness of Moyal parallels the uniqueness theorem of von Neumann, but this goes much further. For any Lie group G with Lie algebra 9 one has an autonomous notion of star representation. Every x E 9 can be considered as a function on g* and restricted to a function u'" E N(W) on a G-orbit (or a collection of orbits) W, so that P(u""uy) realizes the bracket [x, y]g. If we now take a star product on W for which [u""uy]v = P(u""u1l ), what we call a G-covariant star product, the map x t-+ ~v-Iu", will define a representation of the enveloping algebra U(9) in N[v- I , vll, the space of formal series in v and V-I (polynomial in the latter) with coefficients in N, endowed with the star product. This will give a representation of Gin N[[v-1, vll by the star exponential:

00

G 3 e'" t-+ E(e"') = Exp(x) = 2:)n!)-I(u",/2v)*n. (10) n=O

The star product is called G-invariant if [u""v]v = P(u", , v) "Iv E N, i.e., if the geometric action of G on W defines an automorphism of the star product. A whole theory of star representations has been developed (see e.g. [27] for an early review). By now it includes an autonomous development of nilpotent and solvable groups (in a way adapted to the Plancherel formula), with a correspondence (via star polarizations) with the usual Kirillov and Kostant theories [28]; there the orbits are (in the simply connected case) symplectomorphic to some R,2l, and the Moyal product can be lifted to the orbits. Star representations have also been obtained for compact groups and for several series of representations of semi-simple Lie groups [29] (including the holomorphic discrete series and some with unipotent orbits); the cochains Cr are here in general pseudodifferential. Integration over W of the star exponential (a kind of trace) will give a scalar-valued distribution on G that is nothing but the character of the representation.

3. Closed star products

3.1 Trace and closed star products. Existence.

a. For Moyal product (weight w = 1 in (9» one has, whenever f21(U) is trace-class:

(11)

while for other orderings (like the standard ordering S) this formula is true

250 M. Flato and D. Sternheimer

only modulo higher powers of Ii: Tr(ns(u» = TM(U) + O(lil-i). But for all of them the above-defined TM has the property of a trace, i.e.,

(12)

One even has (for the Moyal product, because of the skew-symmetry of the Aij ) that TM(U *M v) = TM(UV). All these star products are what we call [16] strongly closed:

'Vr and u,v E N. (13)

b. The existence proofs of [24] can be made such that the star prod­ucts constructed are strongly closed. When b2 = 0, the uniqueness (modulo equivalence) of star products shows that all d.n.c. star products are equiv­alent to a closed one (which exists). This is true on a general symplectic manifold: All differentiable null on constants star products are, up to equiv­alence, strongly closed.

This is somewhat related to a recent result by O. Mathieu [30] that gives an often (not always, but always in degree 2) satisfied necessary and suffi­cient condition for the existence of harmonic forms on compact symplectic manifolds, and thereby counterexamples to a conjecture by J. L. Brylinski.

To prove this result one considers the algebra N[[v]] endowed with star product and restricts it (in order to get finite integrals) to V[[v]], where V denotes the Coo functions with compact support on the manifold W. A trace on V[[v]] is then defined as a C[[v]]-linear map T into C[v-1 , v]] satisfying (12). In the d. n. c. case it has been shown by Tsygan and Nest [31] that there exists (up to a factor) a unique trace on V[[v]].

If one takes a locally finite covering of W by Darboux charts U Ot (all the intersections of which are either empty or diffeomorphic to ]R21) and a partition of unity (Po) subordinate to it, this trace can be defined, in a consistent way [31], by

(14) Ot

where To. is the Moyal trace (11) on the image of Uo in a standard ]R2l by coordinate maps, and To an equivalence that maps the given star product restricted to UOt into the Moyal product of the standard ]R2i (any self­equivalence of the Moyal product on ]R2i preserves the Moyal trace). It is

Closedness of Star Products and Cohomologies 251

thus of the form

00

with T = 1+ LvrTr, (15) r=l

the Tr being differential operators, and this T transforms the initial product into an equivalent one that is obviously closed.

c. In the last construction the d.n.c. assumption is not necessary, but relaxing it is not a trivial matter because it involves going beyond the differentiable case for the Hochschild cohomology and (for the n.c. assumption) because it is needed that 1 be a unity for the star algebra (differentiable star products can however be made d.n.c. by equivalence). For more general star products the difference between general, closed and strongly closed star products should become manifest.

A star product is said to be closed if (13) is supposed only for r ~ i, i.e., if the coefficient of vi in (u * v - v * u) for all u, v E V[[vll has a vanishing integral. The reason for this definition is obvious from a glance at (11) if one remembers that in the algebraic index theorems of A. Connes [14], indices of operators are expressed as traces, and that star products permit an alternative and autonomous treatment of operator algebras directly on phase-space. Note that all star products on 2-dimensional manifolds are closed (because of (5)).

3.2. Closed star products, cyclic cohomology and index theorems.

a. Cyclic cohomology was introduced by A. Connes in connection with trace formulas for operators (the dual theory of cyclic homology was developed independently and in a different context by B. Tsygan) [32]. The motivation is that cyclic cocycles are higher analogues of traces and thus make easier the computation of the index as the trace of some operator by giving it an algebraic setting.

Let A be an algebra, A = V[[vll to fix ideas (but the notion of cyclic cohomology can be defined abstractly). To every u E A one can associate it E A* defined by

(16)

A acts on A* by (x¢y)(v) = ¢(yvx), with ¢ E A* and V,X,y E A. The map u t-+ it defines a map CP(A, A) -+ CP(A, A*) compatible with the (Hochschild) coboundary operator b, and we can restrict to the space of cyclic cochains Cr c CP(A,A*), those that satisfy the cyclicity condition

252 M. Flato and D. Sternheimer

The cyclic cohomology of A, HCP(A), is defined as the cohomology of the complex (Cr, b). Now, on the bicomplex cn,m = cn-m(A, A*) for n ;::: m (defined as {O} for n < m), b is of degree 1 and one can define another operation B of degree -1 that anticommutes with it (bB = - Bb, B2 = o = b2 ). We refer to [14] and [16] for a precise definition in the general casej when ')'(lj u, v) = J C(u, v)wi is a normalized 2-cochain ')' = 6 E

C 2(A,A*), B can be defined by B')'(u,v) = ')'(lju,v) - ')'(ljv,u). This bicomplex permits to compute the cyclic cohomology (at each level p) by :

HCP(A) = (kerb n ker B)/b(ker B). (18)

Obviously the closedness condition at order 2 of a star product (4,5) is expressed by B62 = o. By standard deformation theory [11] we know that the Hochschild 2-cocycle C1 determines a 3-cocycle E2 that has to be of the form bC2 (if the deformation extends to order 2) and therefore (if the star product is closed at order 2) E2 E kerbnkerB. Since modifying 62 by an element of ker B will not affect closedness, and since [11] the same story can be shifted step by step to any order, (18) shows us that the obstructions to existence of a Cr yielding a star product closed at order r ;::: 2 are given by HC3(A). Similarly, like in [11], the obstructions to extend an equivalence of closed star products from one step to the next are classified by HC2(A). Cyclic cohomology replaces Hochschild cohomology for closed star products, and this will become especially important when the n.c. assumption is not satisfied.

h. Character, and index theorems. As in the Banach algebra case [33], which is a specification of the general framework developed by A. Connes [14], we can define here, when * is closed:

where £ ::; 2k ::; 2£ (otherwise it is necessarily 0),

00

O(Ul, U2) = U1 * U2 - U1U2 = L vrcr(Ul, U2) (20) r=1

is a quasi-homomorphism (that measures the noncommutativity of the *­algebra and is also a Hochschild 2-cocycle) and T is the trace defined by

00

U = L vrur E V[[vll· (21) r=O

Closedness of Star Products and Cohomologies 253

This defines the components of a cyclic cocycle cp in the (b, B) bicomplex on 'D that is called the character of the closed star product. In particular

(22)

When 2l = 4, a simple computation shows that the other component is CP2 = O2 and then bCP2 = -~Bcp4. But HC2(V) = Z2(W,C)E9C (where Z2 denotes the closed 2-dimensional currents on W). Therefore the integrality condition < cP, Ko(V) >c Z, necessary to have a deformation of V to the algebra of compact operators in a Hilbert space, what is traditionally called a quantization, has no reason to be true for a general closed star product.

Now the pseudodifferential calculus on W = T* X, with X compact Riemannian, gives [16] a closed star product, the character of which co­incides with the character given by the trace on pseudodifferential opera­tors, and therefore satisfies the integrality condition (up to a factor). The Atiyah-Singer index formula can then be recovered in an autonomous man­ner also in the star product formulation [14, 16, 23]. But the algebraic index formulas are valid in a much more general context [14, 31, 33]. It is there­fore natural to expect that the character of a general star product should make it possible to define a continuous index.

Recently a number of preprints have appeared (see e.g., [31] and [23]) deriving various proofs and generalizations of the Atiyah-Singer theorems using the (closed) star product formalism.

4. Star products and quantum groups

4.1. Topological algebras. The notion of quantum group has two dual aspects: the modification of commutation relations in Lie and enveloping algebras, and deformations of algebras of functions on a group (with star products). The latter gives by duality a coproduct deformation. In both cases Hopf algebra structures are considered, but there is a catch: except for finite-dimensional algebras, the algebraic dual of a Hopf algebra is not a Hopf algebra. The best way to circumvent this (largely overlooked) dif­ficulty is to topologize the algebras in a proper way.

a. Deformations revisited. Let A be a topological algebra. By this we mean an associative, Lie or Hopf algebra, or a bialgebra, endowed with a locally convex topology for which all needed algebraic laws are continuous. For simplicity we fix the base field to be the complex numbers cr. Extending it to the ring cr[[vll gives the module A = A[[vll, on which we can consider various algebraic structures.

254 M. Flato and D. Sternheimer

A deformation of an algebra A is a topologically free q[vlJ-algebra A such that A/vA ~ A. For associative or Lie algebras this means that there exists a new product or bracket satisfying (4) or (1) (resp.). For a bialgebra (associative algebra A with coproduct ~ : A - A®A and the obvious com­patiblity relations), denoting by ®" the tensor product of q[vlJ-modules, one can identify A®"A with (A®A)[[vlJ, where ® denotes the algebraic ten­sor product completed with respect to some operator topology (projective for Frechet nuclear topology e.g.), we similarly have a deformed coproduct

00

.i = ~+ LvrDr , Dr E C(A,A®A) (23) r=l

and here also an appropriate cohomology can be introduced [34-36]. In the case of Hopf algebras, the deformed algebras will have the same unit and counit, but in general not the same antipode. As in the algebraic theory [12], equivalence of deformations has to be understood here as isomorphism of C[[vll-topological algebras (the isomorphism being the identity in degree o in v), and a deformation is said trivial if it is equivalent to that obtained by base field extension.

b. The required objects. In the beginning Kulish and Reshetikhin [37] discovered a strange modification of the g = 8£(2) Lie algebra, where the commutation relation of the two nilpotent generators is a sine in the semi-simple generator instead of being a multiple of it, which requires some completion of the enveloping algebra Ucg). This was developed in the first half of the 80's by the Leningrad school of L. Faddeev, systematized by V. Drinfeld who developed the Hopf algebraic context and coined the extremely effective (though somewhat misleading) term of quantum group [15], and from the enveloping algebra point of view by Jimbo [38]. Shortly afterwards, Woronowicz [39] realized these models in the context of the noncommutative geometry of Alain Connes [14] by matrix pseudogroups, with coefficients in C* algebras satisfying some relations.

Let us take (for simplicity) a Poisson Lie group, a Lie group G with compatible Poisson structure i.e., a Poisson bracket P on N = Coo(G), con­sidered as a bialgebra with coproduct defined by ~u(g, g') = u(gg'), g, g' E

G, and satisfying

~P(u, v) = P(~u, ~v) where u, v E N. (24)

The enveloping algebra Ucg) can be identified with distributions with (com­pact) support at the identity of G, this is part of the topological dual N' of N. But we shall need a space bigger than N' for the quantized universal

Closedness of Star Products and Cohomologies 255

enveloping algebra U,AQ) , to include some infinite series in the Dirac 8 and its derivatives. Thus shall have to restrict to a subalgebra H of N. When G is compact we shall take the space H of G-finite vectors for the regular representation.

It is natural to look for topologies [40] such that both aspects will be in full duality, i.e., reflexive topologies. We also would like to avoid having too many problems with tensor product topologies that can be quite intricate; for instance, we need to identify COO(G x G) = N®N.

When G is a general Lie group, N is Frechet nuclear with dual &', the distributions with compact support, but V (with dual V', all distributions) is only a (LF)-space, also nuclear. There is no simple candidate to replace the G-finite vectors of the compact case (the most likely are the analytic vectors for the regular, or a quasi-regular, representation). In the following we shall thus from now on restrict to the original setting of G compact.

4.2. Compact topological quantum groups [35].

a. The classical objects. For a compact Lie group G we shall consider the following topological bialgebras (in fact, Hopf algebras):

and its dual H' = II C(Vp) ::) V'.

pEG

(25)

Here Vp is the isotypic component of type pEG in the Peter-Weyl de­composition of the (left or right) regular representation of Gin L2(G). H is also called the space of coefficients, since it is the space of all coeffi­cients of unitary irreducible representations (each isotypic representation being counted with its multiplicity, equal to its dimension). The enveloping algebra U = U(Q) is imbedded in H' by

U 3 x 1--+ i(x) = (p(x» E H'. (26)

This imbedding has a dense image in H'; the image is in fact in V' but is not dense for the V' topology. The product in V' is the convolution of distributions, the coproduct satisfies 6.(g) = 9 ® 9 for 9 E G (identified with the Dirac 8 at g), and the counit is the trivial representation. Since the objects will be the same in the "deformed" case, only some composition laws being modified (exactly like in deformation quantization, of which it is in fact an example), we have discovered the initial group G "hidden" (like a hidden classical variable) in the compact quantum groups.

All these algebras are what we call well-behaved: the underlying topo­logical vector spaces are nuclear and either Frechet or (DF). The impor­tance of this notion comes from the fact that the dual A' of a well-behaved A is well-behaved, and the bidual A" = A.

256 M. Flato and D. Stemheimer

h. The deformations. First let us mention that duality and de­formations work very well in the setting of well-behaved algebras: if A is a deformation of A well-behaved, its C[[vll-dual A~ is a deformation of A * and two deformations are equivalent iff their duals are equivalent deformations.

In view of the known models of quantum groups, we select a special type of bialgebra deformations, those we call [34,35] preferred: deformations of N or H with unchanged coproduct. Here this is not a real restriction because any coassociative deformation of H or N can (by equivalence) be made preferred (with a quasi-cocommutative and quasi-associative prod­uct).

This follows by duality from the fact that for V' or H', any associa­tive algebra deformation is trivial, that these bialgebras are rigid (in the bialgebra category) and that any associative bialgebra deformation with unchanged product has coproduct Li and antipode § obtained from the undeformed structures by a similitude (expressing the "quasi-" properties):

Li = PtlP-l, § = asa-1 (27)

for some P E (A®A)[[vll and a E A [[v]] , with A = V' or H'. When associative, the product is a star product that can (by equivalence [41]) be transformed into a (noninvariant) star product *' satisfying tl(u *' v) = tlu *' tlv, '11., V E N.

This general framework can be adapted to the various models. We refer to [35] for a thorough discussion. The Drinfeld and Faddeev-Reshetikhin­Takhtajan models fall exactly into this framework. An essential tool is the Drinfeld isomorphisms <p, (algebra) isomorphisms between a Hopf defor­mation U" of U and U[[vll, which give (27). (Two Drinfeld isomorphisms give equivalent preferred deformations of H that extend to preferred Hopf deformations of N).

The Jimbo models [38] are somewhat aside because their classical limit is not U but U(Q) extended by Rank( G) parities, and this makes out of the deformed algebras nontrivial deformations.

Finally we are now in position to have a good formulation of the quantum double [42]. If A (resp. A') denote H' or V' (resp. H or N) or their deformed versions, the double is D(A) = A'i8lAj its dual is D(A)' = A®A', D(A)" = D(A) and all these algebras are rigid.

c. Remark. The above procedure can be adapted to noncompact quantum groups. A first step in this direction can be found in [43].

Acknowledgements. The authors want to thank A. Connes, S. Gutt, P. Lecomte, G. Pinczon and B. Tsygan for worthy contributions and A. Wein-

Closedness of Star Products and Cohomologies 257

stein for useful comments. They also thank Ranee and Jean-Luc Brylinski and Ann Kostant, without whom the conference would never have been possible and this volume, hence this paper, would never have appeared.

References

[I} P.A.M. Dirac, Lectures on Quantum Mechanics, Belfer Graduate School of Sciences Monograph Series No.2 (Yeshiva University, New York, 1964).

[2} M. Flato, A. Lichnerowicz, and D. Sternheimer, J. Math. Phys. 17 (1976), 1754-1762.

[3} H. Weyl, The theory of groups and quantum mechanics, Dover, New­York, 1931.

[4} B. Kostant, Quantization and unitary representations, In : Lect. Notes in Math. 170(1970), Springer-Verlag, Berlin, 87-208. J.M. Souriau, Structure des systemes dynamiques, Dunod, Paris, 1970.

[5} M. Flato, Czech. J. Phys. B32 (1982),472-475. [6) M. Flato, G. Pinczon, and J. Simon, Ann. Sc. Ec. Norm. Sup. 4e serie,

10 (1977), 405-418. [7} M. Flato, A. Lichnerowicz, and D. Sternheimer, C. R. Acad. Sc. Paris

A 279 (1974), 877-881; Compositio Mathematica 31 (1975), 47-82. [8} M. Flato and D. Sternheimer, Deformations of Poisson brackets ... in:

Harmonic Analysis and Representations of Semi-Simple Lie Groups, J.A. Wolf et al. (eds.), 385-448, MPAM, D. Reidel, Dordrecht (1980).

[9} J. Vey, Comment. Math. Helv. 50 (1975), 421-454. [1O} M. Flato, A. Lichnerowicz, and D. Sternheimer, C.R. Acad. Sc. Paris,

A 283(1976), 19-24 . [11} F. Bayen, M. Flato, C. Fronsdal, A. Lichnerowicz, and D. Sternheimer,

Ann. Phys. (N. Y.), 111 (1978), 61-110 and 111-151. [12} M. Gerstenhaber, Ann. Math. 79 (1964), 59-90. [13} F. Berezin, Math. USSR lzv. 8 (1974), 1109-1165. [14} A. Connes. Publ. Math. IHES 62, 41-144 (1986); Geometrie Non­

Commutative, Intereditions, Paris (1990) [expanded English edition, preprint IHES/M/93/12, March 1993, to be published by Academic Press} .

[15} V.G. Drinfeld, Quantum Groups in: Proc. lCM, Berkeley, 1, 101-110, Amer. Math. Soc., Providence (1987).

[16} A. Connes, M. Flato and D. Sternheimer, Lett. Math. Phys. 24 (1992), 1-12.

[17} J. Dito, Lett. Math. Phys. 27 (1993), 73-80 and 20 (1992), 125-134; and J. Math. Phys. 33 (1993), 791-801.

258 M. Flato and D. Sternheimer

[18] I.E. Segal, Symposia Mathematica (1974), 14, 79-117. B. Kostant, ibid., 139-152.

[19] S. Gutt, Deformations formelles de l'algebre des fonctions differentiables sur une variete symplectique, Thesis, Bruxelles (1980); Lett. Math. Phys. 3(1979), 297-309; Ann. Inst. Henri Poincare A 32 (1980), 1-31, and (with M. De Wilde and P. Lecomte) A 40 (1984), 77-89.

[20] G. Hochschild, B. Kostant, and A. Rosenberg, 'I'rans. Am. Math. Soc. 102 (1962), 383-406.

[21] J.E. Moyal, Proc. Cambridge Phil. Soc. 45 (1949), 99-124. A. Groenewold, Physica 12 (1946),405-460.

[22] A. Lichnerowicz, Ann. Inst. Fourier Grenoble, 32 (1982), 157-209. [23] B. Fedosov, J. Diff. Geom. (in press) and pp. 129-136 in "Some topics

of modern mathematics and their applications to problems of mathe­matical physics" (in Russian, 1985). See also: Sov. Phys. Dokl. 34 (1989), 319-321; Advances in Partial Differential Equations (in press); Proof of the Index Theorem for Deformation Quantization, (preprint Potsdam Max-Planck-Arbeitsgruppe, February 1994). C. Emmrich and A. Weinstein (Berkeley preprint, Nov. 1993); S. Gutt, Lectures at ICTP Workshop (Trieste, March 1993).

[24] H. Omori, Y. Maeda, and A. Yoshioka, Lett. Math. Phys. 26 (1992), 285-294; Adv. in Math. 85 (1991), 225-255. M. De Wilde and P. Lecomte, Note di Mat. X (1992); Lett. Math. Phys. 7 (1983), 487-496.

[25] M. Cahen, M. Flato, S. Gutt, and D. Sternheimer, J. Geom. Phys. 2 (1985), 35-48.

[26] J. Schwinger, Proc. Natl. Acad. Sci. 46 (1960), 1401-1415. [27] D. Sternheimer, Sem. Math. Sup. Montreal 102 (1986), 260-293. [28] (a) D. Arnal and J.C. Cortet, J. Geom. Phys. 2 (1985), 83-116; J.

Punct. Anal. 92 (1990), 103-135. (b) D. Arnal, J.C. Cortet and J. Ludwig, Moyal product and represen­tations of solvable Lie groups (to be published in J. Funct. Anal.). (c) D. Arnal, J.C. Cortet, and J. Ludwig, Moyal product and represen­tations of solvable Lie groups (preprint, December 1993).

[29] D. Arnal, M. Cahen, and S. Gutt, Bull. Soc. Math. Belg. 41 (1989), 207-227.

[3~] O. Mathieu, Harmonic cohomology classes of symplectic manifolds (preprint, July 1993, to be published in Comment. Math. Helv.).

[31] R. Nest and B. Tsygan, Algebraic index theorem, to appear in Com­mun. Math. Phys. and, Algebraic index theorem for families, to appear in Adv. in Math. B. Tsygan (private communication).

[32] A. Connes, in: Math. Forschunginstitut Oberwolfach Tatungsbericht

Closedness of Star Products and Cohomologies

41/81, Funktionalanalysis und C*-Algebren, 27-9/3-10 (1981). B. Tsygan, Russ. Math. Surveys 38 (1983), 198-199.

259

[33] A.Connes, M. Gromov, and H. Moscovici, C.R. Acad. Sci. Paris I 310 (1990), 273-277. A. Connes and H. Moscovici, Topology, 20 (1990), 345-388.

[34] M. Gerstenhaber and S.D. Schack, Proc. Nat. A cad. Sci. USA, 87 (1990), 478-481; Contemp. Math. 134 (1992), 51-92.

[35] P. Bonneau, M. Flato, M. Gerstenhaber, and G. Pinczon, Commun. Math. Phys. 161 (1994), 125-156.

[36] P. Bonneau, Lett. Math. Phys. 26 (1992), 277-280. [37] P.P. Kulish and N.Y. Reshetikhin, J. Soviet Math. 23 (1983), 24-35.

(In Russian, Zap. Nauch. Sem. LOMI, 101 1981, 101-110). [38] M. Jimbo, Lett. Math. Phys. 10 (1985), 63-69. [39] S.L. Woronowicz, Commun. Math. Phys. 111 (1987),613-665. [40] F. Treves, Topological Vector Spaces, Distributions and Kernels, Aca­

demic Press, 1967. D. Sternheimer, Basic Notions in Topological Vector Spaces, in: Proc. Adv. Summer Inst. in Math. Phys. (Istanbul 1970, A. Barut ed.), 1-51, Studies in Math. Phys., D. Reidel, Dordrecht (1973).

[41] L. Takhtajan, Introduction to quantum groups in: Springer Lecture Notes in Physics, 370 (1990), 3-28.

[42] P. Bonneau, Reviews in Math. Phys. 6 (1994), 305-318. [43] F. Bidegain and G. Pinczon, Lett. Math. Phys. (to be published).

Physique Mathematique, Universite de Bourgogne, B.P. 138, F-21004 Dijon Cedex, France

Received February 25, 1994

The Algebra of Chern-Simons Classes, the Poisson Bracket on it,

and the Action of the Gauge Group

Israel M. Gelfand and Mikhail M. Smirnov

Abstract

Developing ideas offormal geometry [GI), [GKF), and ideas based on combinato­rial formulas for characteristic classes, we introduce the algebraic structure that models the case of N connections given on the vector bundle over an oriented manifold. First we construct a graded free associative algebra A with a differ­ential d. Then we go to the space V of cyclic words of A. Certain elements of V correspond to the secondary characteristic classes associated to k connections. That construction allows us to give easily the explicit formulas for some known secondary classes and to construct the new ones. The space V has new opera­tions: it is a graded Lie algebra with respect to the Poisson bracket. There is an analogy between our algebra and the Kontsevich version of the noncommuta­tive symplectic geometry. We consider then an algebraic model of the action of the gauge group. We describe how elements of our algebra corresponding to the secondary characteristic classes change under this action.

O. Introduction

The problem of localization of topological invariants by the methods of for­mal geometry originated in the combinatorial formulas for characteristic classes (Gabrielov-Gelfand-Losik [GGL)). Closely related to combinato­rial formulas for characteristic classes is the problem of explicit description of secondary characteristic classes and difference cocycles (Chern-Simons [ChS], Cheeger, Bott-Shulman-Stasheff [BSS], Youssin [You] and others).

The idea of using the local data (fields of geometric objects) and a construction of a universal field and a variational bicomplex goes back to [G1] (see also the work of Gelfand, Kazhdan and Fuks [GKF)).

We introduce here the algebraic structures which model a certain differ­ential geometric situation. Let G be a Lie group with finitely many compo­nents and let 9 be its Lie algebra (we shall take G = GL(n, C), or GL(n,R) so 9 consists of all n x n matrices). Let X be a Coo oriented manifold and let Ea be a principal G bundle over X provided with N connections '\11, ... , '\1 N, (N =f n). Let us associate to Ea an n-dimensional vector bun-

This work was partially supported by NSF grant DMS 92-13357.

262 I. M. Gelfand and M. M. Smirnov

dIe 7r : E -+ X and connections wo(x), WI(X), ... , WN(X) on this bundle given by their matrices of I-forms wo(x), WI(X), ... ,WN(X) E OI(X) ® g. Let ill be a k-dimensional simplex with vertices i o, ... ,ik C {O, 1, ... ,N}, ill ~ {to + h + ... + tk = 1, to, ... , tk > O}. Consider a connection wet), t E ill such that wet) = tOWio + ... + tkWik. Then we define a secondary characteristic class

where d is the total differential (with respect to x and t on ill x X [GGLJ) and in the expression under the integral we take only summands with k dt's and forget about the other summands.

Then the following relation holds:

k

ch~(wio·· .Wik) = 2)-I)Pch~-1(Wio .. . Wi p ••• Wik). p=l

We are interested in the explicit expressions for ch~ and the relations and algebraic operations on them.

In the works of Gabrielov, Gelfand and Losik [GGL] and Gabrielov, Gelfand and Fuks [GGF] two double complexes were constructed. One was called the variational bicomplex and the other was called the difference bicomplex. On the other hand, Shulman IS], Bott [B] and Bott, Shulman and Stasheff [BSS] in a slightly different context constructed a bicomplex very similar to the difference bicomplex. Secondary classes appear in a natural way in these complexes.

It is known that the Chern-Simons class constructed from two connec­tions Wo and WI has the form ch~(WO,WI) = !tr((wIdwI + ~wV - (wodwo + ~w5)+d(WIWO»' where Wi are matrices of I-forms which define connections. For higher secondary classes the calculations give longer polynomials. For example

ch~(wo,wd = !tr[(la~ + !aIbIbI + !a~bI)- (lag + !aobobo + !a~bo)+ +id((aoal - aIao)(bo + bI) - (a5 + ai + !aIao)alao)J,

where ai = Wi and bi = dwi . So, even in the simplest case of normal sec­ondary characteristic classes formulas are lengthy and we have to handle combinatorics of long noncommutative expressions. In this section we in­troduce algebraic structures which are useful in calculations, but which are also of independent interest. When we translate the situation to the alge­braic language we can see certain new things that are hard to see on the differential-geometric side. Namely there is a remarkable parallel between

The Algebra of Chern-Simons Classes 263

the algebraic structure which is described below and the Kontsevich version of the noncommutative symplectic geometry [Kon]. We hope to return to this parallel later. Vaguely we can view the algebra A described in the next .... section as a 'cotangent bundle' of its subalgebra generated by a's.

Now let us draw a parallel between the algebraic and the geometric pictures.

GEOMETRY ALGEBRA

connections Wi generators ai of the free algebra A

dJ.ui generators bi of the free algebra A

tr Space of cyclic words V

d d

Curvatures Ri = dJ.ui + w't (bi + an E A

Chern forms Chm(Wi) = trRj" (bi +anm E V

Gauge transformation a, a-1 Generators x, y in the gauge algebra G

daa- 1 c

W -; a-1da + a-1wa a -; y(a + c)x

Oi

Poisson bracket{ , }

etc. etc.

Apparently there are connections between the algebra introduced here and BRST (see [LZ1]) which we do not discuss here.

264 /. M. Gelfand and M. M. Smimov

1. The associative algebra and its cyclic space

Consider an assocJative noncommutative algebra A over C freely generated by N generators

ai, i = 1, ... ,N, of degree 1

and N generators

bi , i = 1, ... , N, of degree 2.

Consider an operation d defined on generators by

dai = bi , i = 1, ... ,Nj

dbi = 0, i = 1, ... , Nj

which is extended to monomials in A by the rule d(PQ) = (dP)Q + (-1)IPIPdQ and then to A by linearity. Here P and Q are monomials in ai, bi and IFI is the degree of P, i.e., the sum of degrees of all letters in P.

Example 1. Let A be generated by 2 letters a and b. Then the elements of A are noncommutative polynomials in a and b. For example, L = 5aaba + aaa + 3bb is an element of A. Then dL = 5((da)aba +(-1)la(da)ba+ (-1)2aa(db)a + (-1)4aab(da» +((da)aa + (-1)a(da)a + (-1)2aa(da» + 3((db) . b + (_1)2b(db» = 5(baba - abba + aabb) + (baa - abaaab).

Definition 1. Consider the factor space V = A/{PQ - (-l)IPIIQIQP}, where {PQ - (-1)IPIIQIQP} is the subspace of A spanned by the commu­tators of all monomials in A. The space V is generated by cyclic words in letters ai, bi and we shall call V a cyclic space. We are going to use the symbol ~ for equality in V to distinguish it from the equality in A which we denote by = .

Example 2. Consider a cyclic space V of the algebra A generated by two letters a and b. Let us write its basis:

a, (a2 ~ 0), b, ba ~ ab, a3 , (a4 ~ 0), a2b ~ ba2 ~ -aba, b2, a5 , a3 b ~ a2ba ~ aba2 ~ ba3 , b2a ~ ab2, (a6 ~ 0), a4b ~ -a3 ba ~ ... ~ ba4,

b2a2 ~ -abba ~ a2b2, abab ~ baba ~ 0, etc.

So we have the basic monomials and all other monomials are obtained by a cyclic permutation with an appropriate sign from the basic monomials: a, b, ab, a3 , a2b, b2, a, a3 b, ab2, a4b, b2a2, ...

The Algebra of Chern-Simons Classes 265

Example 3. In the cyclic space V of the algebra A generated by 4 letters al,a2 and bbb2,

a¥ go becausealal g (-1)1.3a1al g -alaI.

a1 g 0 because a1 g ala~ g (-1)1.3a~al g -a1. More generally

a~k gO . • alb2alala2bl g (-1)1.7b2alala2blal = (-1)2.6alala2blalb2 = = (-1)1-7ala2blalb2al = etc.= (_1)2.6 blalb2alala2.

So for a monomial in V we need to know only the cyclic order of letters in it and sometimes it is convenient to write letters in a circle.

Proposition 1.

1. The operator d is a differential in the associative algebra A: dod = O.

2. The restriction of d to the cyclic space V is a well-defined differential and dod g 0 in V. If P go, then dP gO.

Example 4. a2 g 0 because a·a = (-1)1.la·a, da2 = ba-ab g ba-ba = O.

2. Symmetric functions

To write explicit formulas for secondary characteristic classes, we need the special class of symmetric functions in noncommuting variables.

Definition 1. A symmetric function E[pk, QI] in A is a sum of all possible words with k letters P and 1 letters Q.

Example 1. E[PI, Q2] = PQQ + QPQ + QQP. We can replace P and Q by any elements of A, for example

E[(a2)1, (b)2] = a2bb + ba2b + bba2

E[(a2 + b)l, (ab)2] = (a2 + b) (ab)(ab) + (ab)(a2 + b)(ab) + (ab)(ab)(a2 + b)

Definition 2. A symmetric function E[Pfl, ... ,p;;'m] in A is a sum of all possible words with kl letters P b k2 letters P2 , ... , km letters Pm.

Example 2. E[pl,Ql,Rl] = PQR+RPQ+QRP+QPR+PRQ+RQP,

E[p2,Ql,RI] = PPQR+PPRQ+PQPR+PRPQ+PQRP+PRQP+

+QPPR + RPPQ + QPRP + RPQP + QRPP + RQPP.

We can see that it is not important in which order the arguments P, Q, etc. appear in brackets; what matters is only the exponents showing how many times we should take the same letters.

266 I. M. Gelfand and M. M. Smirnov

Example 3. Let :Ep,q = :E[{a2)P,bq] be the sum of all possible words with p letters a2 and q letters b.

:E1,3 = bbba2 + bba2b + ba2bb + a2bbb ~ 4bbba2

:E2,3 ~ 5bbba2a2 + 5bba2ba2

:E3,3 ~ 6bbba2a2a2 + 6bba2ba2a2 + 6bb2a2ba2 + 2ba2ba2ba2

:E3,4 ~ 7bbba2a2a2a2 + 7bba2ba2a2a2 + 7bba2a2ba2a2 + 7bba2a2a2ba2+ 7ba2ba2ba2a2 +- ...

:E3 ,6 ~ 9bbba2a2a2a2a2a2 + ... + 9bba2a2a2a2a2ba2 + 3ba2a2ba2a2ba2a2

3. Other operations in the algebra, Poisson bracket

The remarkable fact is that V possess a Lie algebra structure with respect to a Poisson bracket. So we have a Poisson bracket on the space containing characteristic and secondary characteristic classes.

First we define certain new operations in A. Let A as before be a free associative algebra generated by ai and bi , i = 1, ... ,N.

Definition 1. A partial derivative tz or oz, where z = ai or bi is defined on monomials by the following rule:

1. If the letter z does not appear in the monomial P, then ozP = O.

2. If P = A1ZA2ZA3Z ... ZAk' where Ai are words (may be empty) without the letter z then

C1 = IA11IzA2 ... Akl,

C2 = IA1ZA211zA3 ... Akl, etc.

We take the initial monomial P and look at all appearences of z in it. Then for every letter z we cyclically permute the word so that this letter z becomes the first letter of the permuted word. Then we delete this letter z. The sum of resulting words will be a partial derivative in z.

Example 1. Suppose A is generated by two letters a and b. Then

Oa{aaabab) = ~aabab + {_1)1.7 ~ababa{ -1)2.6~babaa + (-1)5.3~baaab = aabab - ababa + babaa - baaab

ob{aaabab) = {-I)HPabaaa + {-1)2.6paaaba = aaaba - abaaa.

The Algebm of Chern-Simons Classes 267

Remark 1. Even though we call 8z a partial derivative it does not satisfy the Leibniz rule in A. 8a (aab) = ab - ba i= (8a (aa»b + aa8a b.

Proposition 1. The partial derivative is well-defined in the space of cyclic words V. This means that if L = PQ is a representative in A of a cyclic word LEV and L' g L another representative in A of the same cyclic word so that L' = (-l)IPIIQIQP, then 8z L g 8z L', i.e.,

Definition 2. Let P and Q be elements of V. Then their Poisson bracket is

N

{P, Q} g L(8ai P. 8bi Q + (-1)IPIIQI8aiQ· 8bi P). i=l

The elements P and Q are in V. We take any of their preimages in A also denoted by P and Q and apply to them the operators 8ai and 8bi defined in A. Then 8ai P and 8bj Q are elements of A. We multiply them in A and only then we take the corresponding cyclic element which will be independent of decyclization.

Theorem 1. The Poisson bracket is well defined on V and is linear in its arguments. It has the properties

1. I{P,Q}I = IPI + IQI- 3, where IFI is a degree of P.

2. {P,Q} g (-l)IPIIQI{Q,p}.

3. {P, {Q,R}}+(-l)IPI(IQI+IRI}{Q, {R,P}} +(-l)IRIHQ,P}I{R,{P,Q}} g O.

(Z2-graded Jacobi identity).

Definition 3. Define the operator di by the rule diai = bi, dibi = 0, dia; = 0, dib; = 0, and for a monomial P = Xl ••• xp E A,

p

diP = L(-l)lxl",Xk-ll(XI .. . xk-t}(diXk)(Xk+I ... Xp), k=l

Then di is just the part of the differential d which acts only on the letters with index i: d = L di .

i

268 1. M. Gelfand and M. M. Smirnov

Proposition 2.

I. The operator di in the cyclic space V can be written as the bracket with ! b~ and operator d can be written as the bracket with ! (b~ + ... b~ ) ;

dP g H(b~ + ... + b~),P}.

2. The operators di, d and the Poisson bracket are connected by the equality

-d{P,Q} g {dP,Q} + (-I)IPI{p,dQ}.

Proof. Property 1 immediately follows from the definition of the Poisson bracket and the fact that

N

dP g I)i(8a;P). i=l

Property 2 follows from the Jacobi identify for

• An alternative proof can be done using the following lemma:

Lemma 1. The following equalities are true in A

Proposition 3. We can express partial derivatives using the Poisson bracket as

Proposition 4. Define operator dk on V by equality

N

dk P g L bf(8a;P). i=l

The Algebra of Chern-Simons Classes 269

Then

1) dk 0 dk = 0, and d1 = d, where d is our standard differential on V.

2) The operator dk can be written as a Poisson bracket with k!l (bt+1 + + bk+l) . ••• N •

Proof. Same as Proposition 2. • 4. Chern characters

Definition 1. Consider the algebra A generated by two letters a and b and the cyclic image in V of the polynomials ft ( a 2 + b) k • Let us call the corresponding element of V the k-th Chern character and denote it by chk. So chk g (a2 + b)k.

We introduced in A symmetric functions of noncommuting variables I::p,q = OA the sum of all possible words with p letters a2 and q letters b. For example, I::1,2 = a2bb + ba2b + bba2.

Definition 2. Consider the cyclic image in V of the following polynomials in A

1 (1 bk-1 1 't"' + 1 't"' 1 2(k-l») (k_l)!a k + k+1,wl,k-2 k+2,w2,k-3 + ... + 2k-l a

where I::p,q = sum of all possible words with p letters a2 and q letters b. We shall call corresponding elements of V the k-th secondary Chern character (Chern-Simons class) and denote it by chl.

Theorem 1.

1. Chern characters are closed elements of the cyclic space V

2. The differential in V of Chern-Simons class defined by the above formula is the k-th Chern character:

270 I. M. Gelfand and M. M. Smirnov

In the following example we shall write where convenient b ... b for bk etc . ............... k times

Examples. To translate these formulas into the standard physical notation we should take the a = A matrix of I-forms defining the connection, take the b = dA matrix of two forms and put Tr in front of the expression.

1. k = 2

ch~ ~ -ba(~b+~a2) = Hab+~a3), in standard notation ~Tr(AdA+~A3),

d h1 !!: 1 ( 2 + b)2!!: h C2-2a -C2

2. k=3

ch1 !!: 1.a(lb2 + 1 (a2b + ba2) + la4) !!: 1. (ab2 + i!a3b + i!a5) 3 - 2! 3 4 5 - 31 2 5

dch~ ~ if(a2 + b)3 ~ ch3

3. k =4

ch! ~ ifa(lb3 + !(a2bb+ ba2b + bba2) + Ha2a2b+ a2ba2 + ba2a2) + ~a6) ~ ~ :li(ab3 + ~(a3bb + aba2b + abba2) + ~(a5b + a3ba2 + aba4) + ~a7) ~

~ :li(ab3 + ~a3bb + ~aba2b + 2a5b + ~a7). d h1 tr 1 (2 b)4 tr h c 4 =4ja+ =C4

4. k = 5

ch~ ~ :lia(~b4 + i(a2bbb + ba2bb + bba2b + bbba2)+

+Ha2a2bb + a2ba2b + ba2a2b + a2bba2 + ba2ba2 + bba2a2)+

+Ha2a2a2b + a2a2ba2 + a2ba2a2 + ba2a2a2) + ia8 )

!!: 1. (ab4 + 25 a3bbb + Qaba2bb + Qabba2b + 3-5 a5bb + 2-5 a3ba2b+ - 51 6 6 7 7

+¥aba4b + 4s5a5b + ~a8). There are also other beautiful identities in this algebra. Let 8ntn2 ... nk ~ ban1 ban2 b .. . bank. For example, 8 0 ,3 = bbaaa, 8 1,2,0 = babaab.

Proposition 1. k d( 2k-1) tr ~

2k-l a = "-'l,k-l

2 deb 2k-l) tr E 2k+1 a = 2,k-2

2k~3d[(k -I)80,2k-5 + I81,2k-6 + (k - 2)82,2k-7 + 283,2k-8 + ... J ~ Ek- 3,3·

Example 1. Let 8 p ,q = baPbaq •

E2,3 ~ 5bbba2a2 + 5bba2ba2 ~ ¥d(3 . bba5 + 1 . baba4 + 2ba2ba3)

E3,3 ~ ~d(4. bba7 + 1 . baba6 + 3ba2ba5 + 2ba3ba4)

The Algebra of Chern-Simons Classes 271

E3,4 g; 171 d(5bba9 + 1baba8 + 4ba2ba7 + 2ba3ba6 + 3ba4ba5 )

E3,5 g; 183d(6. 80,11 + 1 . 81,10 + 5.82,9 + 2·83,8 + 4.84,7 + 3.85,6)

5. Algebra with dependence on t and secondary characteristic classes

Let A be a free associative algebra generated by elements a(t), aCt) of degree 1 and elements bet), bet) of degree 2, where t E [0,1]. This algebra has a differential d such that

da{t) = bet), da(t) = bet), db(t) = 0, db = o. This algebra has also a differential (j = dt such that

dta(t) = dta(t) and dtb(t) = dtb(t),

and a derivative in t for expressions in a, b. We denote this derivative by the dot. We are not going to apply dt to expressions with a and b (in other cases we should be introducing letters with more dots but we don't want to deal with them).

From this algebra, as before, we construct a factor space V = AI {C B -(-1)IC IIBIBC} where {CB - {-1)ICIIBIBC} is the subspace of A generated

by commutators. We call V a space of cyclic words and we denote by g; the equality in V.

Consider a path aCt), 0:::; t :::; 1 connecting ao and al and consider

Now we can write an explicit formula for chl(a(t». We shall use certain symmetric functions of noncommuting variables:

Ep,q = E[(a2 )P, (b)q] =sum of all possible words with p letters a2 and q letters b. For example, E 1,2 = a2bb+ba2b+bba2 • Also we shall use functions Ei,p,q = E[a, (a2)P, (b)q] =sum of all possible words with 1 letter a with p letters a2 and q letters b.

Theorem 1. The secondary characteristic class defined by a path aCt) is given by

chlCa(t»

!!: 1 (( 1 bk-1 + 1 '" + 1 '" 1 2(k-l») 10 - (k-l)! a k k+l ul,k-2 k+2 u2,k-3 + ... + 2k-l a 1

1

+ (k~1)!Jdtd(aUEi,o'k-2+ k!l Ei,1,k-3+ k!2 Ei,2,k-4+· .. + 2k:'1 Ei,k-2,O)) o

272 1. M. Gelfand and M. M. Smirnov

The proof is based on the following lemma:

Lemma 1. The following identities hold in V:

............... id(aE[ci, (a2)O, (b)k-2]) g k;;lcibk- 1 - i abk- 1

k!1 d(aE[ci, (a2)1, (b)k-3]) g k!1 ciE[(a2)1, (b)k-2]_ k!1 E[(a2)1, (b)k-2]

k!2d( aE[ci, (a2)2, (b)k-4]) g fficiE[(a2)2, (b)k-3]_ k!2 E[(a2)2, (b)k-3]

------1-d(aE[ci (a2)k-2 (b)O]) g 2k-2ci(a2)k-l __ 1_ (a2)k-l 2k-l ' , 2k-l 2k-l

Example 1.

1. k = 2

ch~(a(t» g ~(ab+ ~a3{ + 11dtd(~aci) 2. k=3

ch~(a(t» g ~(ab2+~a3b+~a5{ + ;!11 dtd(aHcib+bci)+~(cia2+a2ci») g

!!. .!.(ab2 + ~a3b + ~a5)ll + ! fl dtd(.!(acib - ciab) + .!a3ci) - 3! 2 5 ° 21o 3 2

3. k = 4

ch!(a(t» g ft(ab3 + ~a3bb + ~aba2b + 2a5b + ~a7{

6. Poisson bracket and Chern characters

We have defined secondary characteristic classes chl in V which have the

property dchl g chk and we have written an explicit formula for them in Section 4. Now let us look what happens with the Poisson bracket of two ordinary Chern characters and the bracket of an ordinary and secondary Chern characters.

Theorem 1. The bracket of two ordinary Chern characters is zero:

The Algebra of Chern-Simons Classes 273

The Poisson bracket of an ordinary and secondary Chern characters is d­closed.

Proof. Let us notice that 8b(a2 + bt = r~(a2 + b)r-l That follows imme­diately from the definition of 8b. Also,

8a (a2+br = r(~a(a2+br-l-~(a2+br-la) = r(a(a2+br-1-(a2+br-1a).

So {chk,ch,} = {(a2 + b)k, (a2 + b)'} g 8a (a2 + b)k8b(a2 + b)'+

8a (a2 + b)'8b(a2 + b)k g k(a(a2 + b)k-l _ (a2 + b)k-1a)l(a2 + b)'-l+

+l(a(a2 + b)'-l - (a2 + b)'-1a)k(a(a2 + b)k-l) g

g kl(a(a2 + b)k+1-2 _ (a2 + b)k-1a(a2 + b)l-l)+

+kl(a(a2 + b)'-l - (a2 + b)k+I-2 . a(a2 + b)k-l go - 0 = O.

Using the proposition in Section 4 we obtain the d-closeness of the Poisson bracket of an ordinary and secondary Chern characters:

= {chk,ch!} - {chl,O} = 0 - 0 = o. • Let us now write expressions for the bracket of two secondary classes.

First let us take ch~ g ~ (ab + ~a3). Then for a cyclic monomial Q of odd degree

where IIQlla and IIQllb are the numbers of letters a and b in Q. Proposition 1. Let Ep,q be the sum of all possible words with p letters a2

and q letters b. Then

{ 1 } tr 1 ( ch2' aEp,q ="2 q - 2p - l)aEp,q + (p + l)aEp +1,q-l

274 1. M. Gelfand and M. M. Smimov

Proof. The proof is based on the combinatorial fact that

from which we easily get

Example 2. Let us take ch~ g ~a(~Eo,2 + tEI,1 + iE2,O)' Then

{ hI hI}!!:.! (1.:1~ !:i=.!1~ H-4)~ H-7)~) C 2'C 4 - 6 a 2.4L..O,3 + 2.5 L..I,2 + 2.6 L..2,1 + 2.7 L..3,O +

In the general case

tr I (k-2 ~ + ( k-5 + I) ~ + ( k-8 + 2 ) ~ + = (k_l),a 2kL..O,k-1 2(k+1) k L..I,k-2 2(k+2) k+1 L..2,k-3

+ ... + (k2(~';:f + ~)Ep,k-P + ... + O· Ek-I,O)'

We shall write complete formulas for the Poisson bracket in the next paper.

7. Solving the equation dP = Q

Let P(a, b) be an element of a free associative algebra generated by 2 letters a of degree 1 and b of degree 2 the differential d that da = b, db = O.

The Algebra of Chern-Simons Classes

Suppose dP( a, b) = O. Let us take

1

Q(a, b) = J P(ta, dta + tb). o

275

Then dQ(a, b) = pea, b). In the integral above we consider only monomials with dt and we consider dt as an element of degree 1 such that dt·a = -a·dt, d· tb = b . dt and dt . t = t . dt.

Example 1. Let pea, b) = ab - ba. Then d(ab - ba) = b2 - b2 = O.

Q(a, b) = 11 ta(dta + tb) - (dta + tb)ta = 11 (-tdta2 - tdta2 ) = _a2

Now we can formulate a combinatorial algorithm for solving the equa­tion dQ = P, where dP = O. Let P = PI + P2 + ... + Pt, where PI are monomials. For example, let Pi = Xl ....•. Xn, where Xi is one of the letters a or b. For each such monomial we construct a polynomial Qi

a

! Qi = _111 L (_1)8 Xl·· .Xk-l ~ Xk+1" 'Xn,

~ II over all xk=b -------degree=s

where IlPili is the number of letters in the word Pi'

8. Homotopy operators

Now let us construct homotopy operators in the general case.

Theorem 1. Let A be a free associative algebra generated by elements at, .. . ,an of degree 1 and elements bl , ... , bn of degree 2 with differential d s.t. dai = bi for all i = 1, ... ,n.

Then for each P( at, ... , an; bt, ... ,bn) the homotopy operator h defined as

, 1

hP(at, ... , an; bl , ... , bn) = 1 P(tat, ... , tan; (dtal +tbt), ... , (dtan +tbn))

satisfies the identity P = d(hP) + h(dP); in particular if dP = 0 then we can solve equation dQ = P taking Q = hP.

276 I. M. Gelfand and M. M. Smimov

Theorem 1'. The following combinatorial formula holds for hP. Let 'US

write P as a sum of monomials: P = Pi + P2 + ... + Pp • For example, Pi = Xl ••• xp , where Xi is one of a 's or b's. Then

where II~II is the total number of letters in Pi. So we take a word Pi of and replace each letter b with some index * by the letter a with the same index. We give the resulting word the sign (_1)8, where s is the sum of degrees of all letters in front of bk. So we get a sum of IIPII words of degree IPil- 1.

Example 1. Let Pi = baabab. Then

Qi = hPi = !((-1)Oaaabab+ (-1)4baaaab+ (-1)7baabaa+ !(aaabab + baaaab - baabaa).

Proposition 1.

1. The homotopy operator h is a (coJdifferential in the algebra A: hoh = O.

2. The operator h is well-defined on the cyclic space V and is a (co J dif­

ferential there: h 0 hlv ~ 0 and if P ~ 0 then hP ~ O.

3. The operators hand d in A are connected by the formula hod+doh = id; so for every Q E A,

h(dQ) + d(hQ) = Q,

4. The operators hand d in V are connected by the formula

hod + doh g: id

so for every P E V,

h(dP) + d(hP) ~ P.

Example 2. Let P = baba g: 0 because baba = (-1)3.3baba. Then hP = l(aaba + (-1)3baaa) ~ ~(baaa - baaa) = O. Let M = baabab. Then

hM = ~ (aaabab + baaaab - baabaa)

The Algebm of Chern-Simons Classes 277

hhM = i (i ( -aaaaab + aaabaa) + i (aaaaab - baaaaa) - i (aaabaa + baaaaa)) = O.

Let us now define the operators hk for a monomial P = Xl •.. xp by the equality

1 def 1 '"" ()8 U ~ hkPi = IIPil1 .LJ -1 Xl··· Xj-l 'I'k Xj+! ••• X p , h = .LJhk

over all Xj=bk -----d k=l egree=s

Define operators h, hk for a monomial P = Xl ... xp by the equality

1 hP=L

all xj=b.

(-1)SXl ••• Xj_l ~* Xj+l ••. X p , -----degree=8 ak

1 hkI{ = L

all Xj=bk

(_1)8 Xl ... Xj-l ~k Xj+! .•• X p , -----degree=s

Then

(dh + hd)P = IIPIIP-

Proposition 2. The operotors h, hi and the Poisson brocket are connected by the equalities

N

-h{P, Q} g; {hP, Q} + (-l)IPI{p, hQ} - 2 L 8b;P· 8b;Q, i=l

The proof is based on the following lemma.

Lemma 1. The following equalities are true in A:

278 I. M. Gelfand and M. M. Smirnov

Proposition 3. In the cyclic space V, the operators hand h can be written as

Theorem 2. The operators di and hi, i = 1, ... , N satisfy the following identities:

(1) dh + hd = id, where d= Ldi , and h=Lhi

(2) didj + djdi = 0,

(3) hihj + hjhi = 0,

(4) hidj + hjdi = ° for i ¥= j. (5) For an arbitrary monomial PEA

(dihi + hidi)P = '1"li P,

where IlPlli is the number of letters ai and bi in a monomial P, and IIPII is the total number of letters in P. IIPII should not be confused with IPI which is the total degree of P i. e. sum of degrees of all letters in P.

9. Action of the Gauge group and its algebraization

We repeat here the standard basic facts about connections and curvature. Consider a vector bundle E over a base manifold M. Let

be a basis of local sections of E. Let

be another basis. We shall write I,] for a column and (.) for a row vector. Let (f) = (h(x) ... f(x» be coordinates of the section of the bundle written in the basis Ie]. Then the section f can be written as f = (f)le]. We define a connection 'V as an operator on sections satisfying

The Algebra of Chern-Simons Classes 279

for an arbitrary function J.t and an arbitrary section f of E. A connection acts on the basic sections as

And the matrix w of I-forms which defines the connection transforms as follows:

w ~ da . a- l + awa- l .

R =t (dW - w 1\ w) = dw + W 1\ W

is called the curvature tensor. Here w is the matrix of I-forms which is the transpose of w.

A connection 'V can be written as d + w:

'V[!] = d[f] + w[!] and 'V(f) = d(f) + (f)w.

Consider now the gauge transformation a. It transforms w:

where 0' is the transpose of a.

10. Gauge algebra and secondary classes

Let us correspond

to Wi ai,

to dwi bi ,

to O'-l V, to 0' X,

to du- l q, to dO' p.

Consider an associative algebra G generated by letters ai, bi , x, V, p, q, i = I ... N with a differential d such that

dai = bi , dbi = 0,

280

dx=p, dp=O, dy= q, dq=O.

1. M. Gelfand and M. M. Smimov

subject to the relation xy = yx = 1. We shall call G gauge algebm. From the equation xy = yx = 1 we get

d(xy) = d(yx) = 0,

py + xq = 0, qx + yp = 0.

From the equation py + xq = ° we obtain

-p = xqx, -q = ypy.

We can obtain a whole series of identities by further differentiating xy+yx = 1. We shall make much use of the expression py and we introduce a special letter c for it.

because de = d(py) = _pq = _p( _ypy) = (py)2. Elements of the algebra G have the following degrees:

x, yare of degree 0,

p, q, ai are of degree 1,

bi are of degree 2,

e is of degree 1.

The gauge transformation a -+ yp + yax = y(a + c)x transforms

b = da -+ dyp + ydp + dyax + ydax - yadx = y(b + a2 - (a + e)2)x. We

shall consider a cyclic space Va for G: Va = G/[G,Gj. We shall write ~ for equality in Va.

Example 1.

The Algebm of Chern-Simons Classes 281

b ~ dyp + ydp + dyax + ydax - yadx = = qp + qax + ybx - yap = ybx - yap + qax + qp

a2 ~ (yp + yax)(yp + yax) = ypyp + yap + ypyax + ya2x = ya2x + yap - qax - qp

b + a2 ~ y(b + a2)x

So a2 + b behaves as a tensor under the gauge transformation.

Example 2. Let us look at what happens to the element ch~ = !(ab+ ia3) under the gauge transformation.

We can check explicitly that

We know da = b, db = 0, dc = c2.

Theorem 1. Under the gauge transformation in the cyclic space V(C) the Chern-Simons class chi (a) transforms to

hl() h-1 tr 1 ( ) ~ ( )"1 (k + {3 - 1)!'Y! ~[(b)a ( 2)f3 ( )"1] c k a ~ c k = (k-1)! a + c L.....- -1 (k + (3 + )! LA , a , u , a+f3+"I=k-1 'Y

where u = ca + ac + c2 and E[(b)a, (a2)f3, (uP] is the sum of all possible words in C with 0: letters b, f3 letters a2 and'Y letters u.

Proof.

1 11 = dt(a + c)(tb + t2a2 + (t2 - t)u)k-1 (k - 1)! 0

282 I. M. Gelfand and M. M. Smirnov

= 1 ( ) " ( )1' (k + .B - 1) !')'! [()'" ( 2){J ( )1'] (k _ I)! a + e L...J -1 (k +.B + )! E b ,a ,U , ",+{J+I'=k-1 ')'

because

Example 1. Under a gauge transformation the secondary classes ch~(a) and ch~ (a) are transformed in the following way. To translate these formulas to theinost common notation we should take a = A the matrix of I-forms defining the connection and take b = dA the matrix of two forms. For a gauge transformation 9 : A -+ g-1dg + g-1 Ag we take e = dg . g-1. We should also put Tr in front of the whole expression.

So ~Tr(AdA+~A3) -+ ~Tr(AdA+~A3)+~«dg.g-1)dA-(dg.g-1)2A­

i(dg. g-1 )3)

-~[eea + cae + aec]b + ~[eaa - aca + aae))

The expression ~ ( 1~ e5 + ~ac4 - ... ) is d closed in the cyclic space VG. Let us check it.

-~[cca + cae + acc]b + ~[eaa - aca + aae]b) g

The Algebra of Chern-Simons Classes 283

Example 2. (*)Let M be a 3-manifold and 9 : M - G = SU(2) is a smooth mapping. Let >. E 0 3 (G) be a fundamental form. Then

Consider a principal SU(2) bundle over M with connection A E 03(M) ®g. Define

CS(A) = 1M Tr(AdA + ~A3)

The gauge transformation acts on A g: A _ A9 = g-ldg + g-lAg.

But from calculations in our algebra we know that

so

11. Appendix. Explicit formulas for secondary classes ch%

Let ~ be a 2-simplex to + h + t2 = 1, to, tI, t2 > o.

(*) We want to thank Lisa Jeffrey who brought to our attention this example.

284 I. M. Gelfand and M. M. Smirnov

Consider 3 connections ao, at. a2 on the bundle E -+ M. Consider the connection aCt), t E ~ such that aCt) = toao + hal + t2a2. Then

where

and both P and Q are of degree 2. We are interested only in summands with 2 letters P and (k - 2) letters Q because only they will give a 2-form in dt which can be can be integrated over the 2-simplex~. Let us take to = 1 - tl - t2, then dto = -(dtl + dt2). When integrating over ~ we in fact integrate over the triangle 0 ~ tl ~ 1, 0 ~ t2 ~ 1 - tl.

We have

For future calculations we shall use the following integral:

The Algebra of Chern-Simons Classes

Case k = 2,

where Al = al - ao, A2 = a2 - ao·

Case k = 3,

Really

because

{ I! 1 1t:. ti = 3! = 6'

( 2 2! 1 1t:. ti = 4! = 12'

where i,j = 1,2,3 and i =J. j.

Case k = 3 and a2 = 0

h2 ( ) tr 1 J( )3 tr C 3 ao, al, 0 =,. . . . = 3.

285

286 I. M. Gelfand and M. M. Smimov

This coincides with the exact term obtained in the example of section 5:

11 1

ch~(a(t)) g fr(ab2 + ~a3b + ~a5) 0 + 4 fo dt U(aab - aab) + !a3a)

if we take a(t) = tao + (1 - t)al.

12. Appendix. Formulas for the secondary class chl(ao, ad

Now let us write several venient formulas for a secondary class

Theorem.

1 ( tr 1 " 1 '" chk ao,at) = (k_I)!A L.J 2m+s+1,",(1n,B,r) 1n+s+r=k-l

where A = al - ao, R = bo + a~, S = (b1 - bo) + (aoA + Aao), M = A2 and ~(1n,8,r) is the sum of all possible words with m letters M, s letters S and r letters R.

Sketch of the proof.

1

= k\j(dt1 (al-Io) + (bo + a~) + . '----'---­oAR

The Algebra of Chern-Simons Classes 287

!!: ~kA ~ 1 E - k! L....J 2m + s + 1 (m,s,r)'

m+s+r=k-l •

References

[B] R. Bott, On the Chern-Weil homomorphism and the continuous co­homology of Lie groups, Advances in Math. 11 (1973),289-303.

[BR] F. Beresin and V. Retakh, A method of computing characteris­tic classes of vector bundles, Reports on Mathematical Physics 18 (1980), 363-378.

[BSS] R. Bott, H. Shulman, J. Stasheff, On the de Rham theory of certain classifying spaces, Advances in Math. 20 (1976),43-56.

[C] J. Cheeger, Spectral geometry of singular Riemannian spaces, J. Dif­ferential Geom. 18 (1983), 575-657.

[CS] S.S. Chern, J. Simons, Characteristic forms and geometric invariants, Ann. Math. 99 (1974), 48-69.

[Du] J. Dupont, The Dilogarithm as a characteristic class of a flat bundle, J.Pure and Appl. Algebra 44 (1987), 137-164.

[FZ] LB. Frenkel and Y. Zhu, Vertex Operator Algebras Associated to Affine Lie Algebras, Duke Math. J. 66 (1992), 123

[GFK] LM. Gelfand, D.B. Fuks, and D.A. Kazhdan, The actions of infinite dimensional Lie algebras, Funct. Anal. Appl. 6 (1972),9-13.

[GGL] A. Gabrielov, LM. Gelfand, and M. Losik, Combinatorial calculation of characteristic class, FUnktsional Anal. i Prilozhen. (FUnct. Anal. Appl.) 9 no. 2, 12-28 no. 3, 5-26, (1975), 54-55.

[GM] LM. Gelfand and R. MacPherson, A combinatorial formula for the Pontrjagin classes, Bull AMS 26 no .2, 1992, 304-309.

[GTs] LM. Gelfand and B.L. Tsygan, On the localization of topological invariants, Comm. Math. Phys. 146 (1992), 73-90.

[H] F. Hirzebruch, Topological Methods in Algebraic Geometry, Springer­Verlag, Berlin, 1966.

[KLV] LS. Krasil'shchik, V.V. Lychagin, and A.M. Vinogradov, Geometry of jet spaces and nonlinear partial differential equations, Gordon and Breach, NY 1986.

[Kon] M. Kontsevich, Formal (NonJ-Commutative Symplectic Geometry, The Gelfand Mathematical Seminars 1990-92, L.Corwin, LGelfand, and J. Lepowsky, eds., 173-189, Birkhauser, Boston 1993.

[L] J.L. Loday, Cyclic homology, Springer, Berlin, 1992.

288 I. M. Gelfand and M. M. Smirnov

[LQ] J.L. Loday and D. Quillen, Cyclic homology and the Lie algebra of matrices, Topology 24 (1985), 89-95.

[LZl] B.H. Lian and G.J. Zuckerman, New Perspectives on the BRST­Algebraic Structure of String Theory, hep-th / 9211072.

[LZ2] B.H. Lian and G.J. Zuckerman, Some Classical and Quantum Al­gebras, Lie Theory and Geometry, Birkhiiuser, 1994, this volume pp.509-529

[Ml] R. MacPherson, The combinatorial formula of Garielov, Gelfand, and Losik for the first Pontrjagin class, Seminaire Bourbaki No. 497, Lecture Notes in Math. 667, Springer, Heidelberg 1977.

[M2] R. MacPherson, Combinatorial differential manifolds, in: Topolog­ical Methods in Modern Mathematics : A Symposium in Honor of John Milnor's Sixtieth Birthday, Lisa R. Goldberg and Anthony V. Phillips, eds., Publish or Perish, Inc., Houston 1993.

[MQ] V. Mathai and D Quillen, Superconnections, Thorn classes, and equivariant differential forms, Topology 25 (1986), 85-110.

[0] Olver, P.J., Applications of Lie groups to the Partial Differential Equations, Springer, Berlin 1985.

[P] L. Positselskii, Quadratic duality and curvature, Funct. Anal. Appl. 27 (1993), 197-204.

[Q] D. Quillen, Superconnections and the Chern character, Topology 24 no.l, 1985, 89-95.

[Ts] Tsujishita, T., On variation bicomplex associated to differential equa­tions, Osaka J. Math. 19 (1982),311-363.

[You] B.V.Youssin, Sur les formes sp,q apparaissant dans Ie calcul com­binatoire de la deuxieme classe de Pontriaguine par la methode de Gabrielov, Gel'fand et Losik, C.R. Acad.Sci.Paris, t.292, 641-644.

Mathematics Department, Rutgers University, New Brunswick, N.J., 08903, USA.

Mathematics Department, Princeton University, Princeton, N.J., 08544, USA.

Received April 22, 1994

A Distinguished Family of Unitary Representations for the Exceptional Groups of Real Rank = 4

Benedict H. Gross and Nolan R. Wallach

Dedicated to Bert Kostant, with admiration

In this note, we will construct three small unitary representations for each of the four simply-connected exceptional Lie groups G of real rank = 4. We will describe the restrictions of these representations to a maximal compact subgroup K of G, and will show they are multiplicity-free. The method of construction is by a continuation of the "quaternionic discrete series" for G. This works in more generality, and we will treat it fully in another paper, so we have only sketched the proofs here.

The four exceptional groups considered here form a family, indexed by the real division algebras lR, C,!HI, and 0 of dimensions f = 1,2,4, and 8. The disconnected "exceptional" group Spin(4,4) )<183 of type D4 forms the oth-member of this family, corresponding to the lR-algebra of dimension f = O. The structural results we obtain on the quaternionic discrete series and their unitary continuations are all uniform in the family, and extend to the D4 case. In particular, we obtain a new construction of the ladder representation of 80(4,4)+, which was studied by Kostant [K) .

We end with some remarks on the restriction of the ladder representa­tion of G = ES,4 to its symmetric subgroups H, and give a new construction of the ladder representation of the split group G' = Es,s.

The research of both authors was partially supported by NSF summer grants.

1. Groups

Each of the exceptional complex Lie algebras g of type F4 , E6 , E7 , and Es has a unique real form go with real rank = 4. The Satake diagram for go is given on the next page.

In all cases, the real root system of go is of type F4. The 24 long roots a of F4 have eigenspaces go of dimension = 1, and the 24 short roots a of F4 have eigenspaces go of dimension = f = 1,2,4, and 8 respectively [0-V; 315-316]' [H; Table VI, 534].

290 B.H. Gross and N.R. Wallach

F4 o---o~o---o split form

~--------~ Es 0---0---0---0---0

I quasi-split form

o E7 0---0---0---.---0---.

I •

1::8 0 --- • --- • --- • --- 0 --- 0 --- 0

I •

Let G be the simply-connected real Lie group with Lie algebra 90. Then G is a 2-fold covering of the connected group G(JR), where G is the simply-connected algebraic group defined over JR with Lie algebra 90. The maximal compact subgroup K of G has the form

K=SU2 X Mo,

where Mo is the compact, simply-connected simple group of type C3 ,A5 ,D6 , and E7 respectively [O-V; 315-316, 321].

Let t be the complexified Lie algebra of K, and write 9 = t E9 P as a representation of K. Then

where C2 is the standard 2-dimensional representation of SU2, and V is an irreducible, symplectic representation of Mo. We have dim V = 2d, with d = 3/ + 4 = 7,10,16, and 28 respectively. Following the numeration of Bourbaki [B; Planches] the highest weight of V is: W3 for C3, W3 for A5 , W6

for D 6 , and W7 for E 7 • Here is a table:

9 / d= 3/+4 Mo V

F4 1 7 C 3 = SU3,H W3 of dim 14 V E9C6 = A3 C6

E6 2 10 A5 = SU6 W3 of dim 20 V = A3C6

E7 4 16 D6 = Spin12 W6 of dim 32 V = ~ - spin

Es 8 28 E7 W7 of dim 56 V = minuscule

Exceptional Groups of Real Rank =4 291

2. Orbits

The compact Lie group Mo acts linearly on the complex vector space V, so its complexification M acts algebraically on the projective space JP'(V) of hyperplanes in V. (Since V is a self-dual representation of M, the definition of projective space - lines or hyperplanes - is not of great significance. We choose hyperplanes, so that the coordinate algebra of JP'(V) is the symmetric algebra se(V) = EBsn(v) on V, rather than on its dual).

n~O

Proposition 2.1. There are four M-orbits on JI»(V), whose closures form a descending chain of invariant subvarieties:

JP'(V)

I X

I Y

I z

X is a quartic hypersurface, Y = sing(X) is defined by an irreducible mod­ule of cubic relations, and Z = sing(Y) is defined by an irreducible module of quadratic relations. We have

dim X = 2(d - 1) = 6f + 6 dim Y = ~ (d - 1) - 1 = 5 f + 4 dim Z = d - 1 = 3f + 3.

Proof. These orbits, and their stability subgroups, were determined by Segre [Se] when M = SL6 (cf. [D, Ch. 3]), by Igusa [I; Prop. 7, Prop. 3] when M = SP6 and Spin12, and by Harris [Ha; §5] when M = E 7 .

We wish to describe the coordinate algebras of the closures of the or­bits. To do this, fix a positive Weyl chamber for M, and let>. be the highest weight of the irreducible representation V in this chamber. Simi­larly, let J.L be the highest weight of the adjoint representation of M on its Lie algebra m. Finally, let v denote the highest weight of the irreducible representation W of M which occurs in the decomposition: A 2V = W E9 C. The weights >., J.L, and v are linearly independent; here is a table following

292 B.H. Gross and N.R. Wallach

the numeration of Bourbaki [B, Planches).

9 M ). of V J-t of m = Lie(M) v of A 2V - C

F4 Sp6(C) W3 of dim 14 2Wi of dim 21 2W2 of dim 90

E6 SL6(C) W3 of dim 20 (Wi + ws) of dim 35 (W2 + W4) of dim 189

E7 Spin12(C) W6 of dim 32 W2 of dim 66 W4 of dim 495

Es E 7(C) W7 of dim 56 Wi of dim 133 W6 of dim 1539

For a, b, c non-negative integers, we let 1I"(a).+bJ-t+cv) be the irreducible representation of M with highest weight a). + bJ-t + cv. This representation is symplectic if a is odd, and orthogonal if a is even. We shall see that these are the only irreducible M-modules which occur in the decomposition of S-(V) = EBsn(v).

Proposition 2.2. 1) The first four symmetric powers of V decompose into the following irreducible M -modules:

Sl(V) = 11"().)

S2(V) = 11"(2).) 6111"(J-t) S3(V) = 11"(3).) 61 11"(). + J-t) 61 11"().) S4(V) = 11"(4).) 6111"(2), + J-t) 6111"(2J-t) 6111"(v) 6111"(2).) 61 C/4.

2) The algebm of M -invariants in S- (V) is a polynomial ring C[/4) gener­ated by the invariant f4 in degree 4. The algebm S-(V) is isomorphic, as an M -module, to A-(X) ® C[f4), where A-(X) is the coordinate algebm of the hypersurface X defined by the equation (/4 = 0).

3) The ideal of X is genemted by Cf4 in S4(V). The ideal ofY is genemted by 11"().) = (8/4/8v: v E V) in S3(V). The ideal of Z is genemted by 1I"(J-t) in S2(V).

4) The M -module An(x) of elements of degree n in A-(X) is multiplicity­free, and isomorphic to 6111" (a). + bJ-t + cv), where the sum is taken over a, b, c ;::: 0 which satisfy:

{ a + 2b + 4c = n - 2k O::;k::;a

Equivalently, the sum is taken over a, b, c ;::: 0 which satisfy

{ a::n (mod 2) a + 2b + 4c ::; n ::; 3a + 2b + 4c

The M-module An(y) is isomorphic to 6111"(a).+bJ-t), where the sum is taken over a, b ;::: 0 which satisfy a + 2b = n.

The M -module An(z) is isomorphic to the irreducible representation 1I"(n).).

Exceptional Groups of Real Rank =4 293

Proof. The ring of invariants is determined in [S-K, §7), who also give an explicit formula for 14. Since this ring is free on a single generator, S-(V) = A-(X) ® c(14) [Sw; 4.1). The main problem is the determination of the M-module structure of A-(X).

Let N c M be the subgroup which fixes a generic vector v in V (i.e., one whose dual hyperplane (v}.l does not lie on the hypersurface Xc P(V». We have the following table [S-K, §7), [Sw; Prop. 4.10):

g M N

F4 SP6(C) SL3 (C)

E6 SL6(C) SL3 (C) x SL3 (C)

E7 Spin12(C) SL6(C)

Es E7(C) E6(C)

Then the multiplicity of an irreducible M-module W in A-(X) is equal to the dimension of the vector space HomN (W, C) [Sw; Prop. 4.6, Cor. 4. 7J.

We can calculate the space HomN (W, C) using a simple extension of the Cartan-Helgason theorem: it has positive dimension iff W ~ rr(a>.+bp.+cv), in which case it has dimension = a + 1. The rest of the proof is not difficult, and we omit it here. Similar results on S- (V) have been obtained by Brion [Br,p 13J.

From a calculation of the M -orbits on V, we obtain certain nilpotent Kc-orbits on p. We recall that for a general semi-simple connected real Lie group G, the nilpotent Kc-orbits in pep) form a subscheme T defined by the vanishing of all the Kc-invariants in seep). We have S-(p)Kc = C[h,··· ,/dJ, where the number of generators is equal to the real rank r of G, and the degrees of the generators are the degrees of the invariants for the relative Weyl group (cf. [K-RJ). The subscheme T has codimension = r in Pcp).

In our case, the real root system of G is of type F4 , Kc = SL2 (C) x M and I' = C2 ® V. Hence the product variety P(C2 ) X P(V) = p 1 X P(V) maps to Pcp) via the Segre embedding, where it has degree = 2d = dim V. The image is contained in Tj hence the closures of M-orbits X, Y, and Z in P(V) give closures of nilpotent Kc-orbits pl x X, pl X Y, and pl x Z in

294 B.H. Gross and N.R. Wallach

p(~). Here is a picture:

h = 16 = Is = h2 = 0

lP(~)

I T ____

pI X P(V)

I pI X X

pI X Y

pI X Z

The three small unitary representations of G we will construct are unipotent, and associated to the (closures of) Kc-orbits pI X X, pI X Y, and pI X Z in P(~). We note that pI X Z is the unique closed Kc-orbit in P(~ ).

Note. The existence of a unitary representation of G associated to the minimal Kc-orbit in P(~) was first considered, for general real groups, by Vogan (V]. He proved that such a representation (called a ladder represen­tation) existed for split groups of type An, en, F4 , and Es, but did not exist for split groups of type Bn (n ;::: 4). Other constructions of ladder repre­sentations for real groups may be found in (EPWWj and (B-K]. Kazhdan and Savin (K-S] found a construction for split groups of type Dn , E6, E7 ,

and Es which extends to the p-adic case.

3. Representations

We begin by sketching a construction of certain "quaternionic discrete se­ries" representations of G associated to the Kc-subvariety pI X P(V) of P(~), then study their continuations. Let T eKe G be a compact maxi­mal torus, and let ~ etc g be the corresponding Cartan sub-algebra of g. Let b be a Borel sub-algebra of g containing ~. If ¢ = ¢(~, g) = ¢(T, g) are the roots of ~ on g, the choice of b is equivalent to the choice of a positive root system ¢+ C ¢. Let ~ be a root basis for ¢+, and write ~ = ~c U ~n where the roots in ~c are compact (occur in t) and the roots in ~n are non-compact (occur in ~).

We insist that b be chosen so that ~n consists of a single root fro, which has multiplicity two in the highest root. This is possible, and b is

Exceptional Groups of Real Rank =4 295

unique up to conjugation by the compact Weyl group, by a theorem of Borel and de Siebenthal [BdSj Thm. 6]. Let p denote half the sum of the roots in 4>+, and let (3 denote the highest root in 4>+. Then (3 is compact, as multao «(3) = 2. If (3v : 8 1 ---+ T is the co-root associated to (3, conjugation by (3V(-1) defines the Cartan involution (J of g. The Dynkin diagrams of D. are given below, with the vertex corresponding to the unique simple non-compact root 0:0 circled:

F4 e---e~e---@

Es e---e---e---e---e I

@ E7 @--- e --- e --- e --- e --- e

I •

Ea • --- • --- • --- • --- • --- • ---@ I •

Let Qc C G(C) be the standard maximal parabolic subgroup associ­ated to the co-root (3v. The Lie algebra q of this parabolic is the direct sum of ~ and the root spaces ga with ((3v, 0:) 2:: O. We have q = I El1 u, where I = ~ El1 E9 ga is the Levi subalgebra and u is the nilradical

({3v,a)=O

of q. Since ((3v, 0:) = 0,1,2 for all 0: E 4>+, with ((3v, 0:) = 2 only for 0: = (3 [B, VI, 1.8], the algebra u is 2-step nilpotent of Heisenberg type. We have dim(u) = 2d + 1 and Center(u) = [u, u] = g{3 = un t. The quotient uab = u/[u, u] is of dimension 2d, and affords the representation V of the semi-simple part m = [I, I] of L

Let Qc be the opposite parabolic, associated to the coroot - (3v j the quotient G(c)/Qc = V has the structure of a complex projective variety in ]P>(g), with holomorphic tangent space ~ U of dimension = 2d + 1. The real group G --+ G(JR) '--+ G(C) has an open orbit D C V, with stability subgroup L = U1 X Mo c K = 8U2 X Mo. The K-orbit of the unique L-fixed point of D is a compact complex submanifold:

C = K/L '--+ D = G/L.

This realizes D as the "twistor covering" of the quaternionic symmetric space X = G/K, with fibres C ~ ]P>1(C) (cf. tWo]).

A theorem of Schmid and Wolf [S-W] shows that C is the largest com­pact complex submanifold of D, and that, for any coherent analytic sheaf F on D, Hi(D,F) = 0 for all i 2:: 2.

296 B.B. Gross and N.R. Wallach

For the general construction of quaternionic discrete series, one must consider G-equivariant vector bundles on D, but here we will restrict at­tention to the case of G-equivariant line bundles C. These are indexed by elements of the free abelian group Hom(L,C*) = Hom(Ul x Mo,C*) = 7L.(-(3/2). The pull-back of the line bundle Ck = C(k· =!-) to the curve C c:= Pl(C) in D is the line bundle Op(-k). Hence, as a representation of SU2 :

if k:5 1

if k ~ 2

We remark that the line bundle Ck extends to b if and only if k is even, and that HO(b, C-2 ) = HO(b, C«(3» affords the adjoint representation 9 ofG.

Proposition 3.1. Assume that k ~ 2. Then HO(D, Ck) = 0, and HI (D, Ck) affords a smooth infinite-dimensional representation 7rk of G.

The representation 7rk is admissible, of finite length, with infinitesimal character p- ~(3. It has Gelfand-Kirillov dimension = 2d+ 1 and Bernstein degree = 2d. The K -finite vectors in Hl (D, Ck) afford the representation

EB sk-2+n(c2) ® sn(V) of SU2 X Mo. n~O

Finally, there is a unique closed irreducible G-submodule 7r~ of 7rk,

which is the closure of the G-span of the "minimal" K -type Sk-2(C2) ® C.

Proof. This all follows easily from results in Schmid's thesis [S]. In partic­ular, the K -decomposition is obtained by considering the order of vanishing of cohomology classes in a neighborhood of the algebraic curve C '---> D. Since the normal bundle to C is given by the representation C/3/2 ® V of L, the classes vanishing "to exact order n" along C give the K -submodule sk-2+n(c2) ® sn(v).

If a G-submodule W of 7rk not contain the K -type Sk-2(C2) ® C, then the cohomology classes in W vanish along C and all of its G-translates. Since these translates cover D, this forces W = o.

As examples ofthe representations 7rk, the canonical line bundle n(D) of b pulls back to the equivariant bundle C2d+2 on D. The representation 7r2d+2 on HI(D, n(D» is irreducible, and lies in the discrete series for G with infinitesimal character = p = w/3(p - (d + 1)(3). More generally, 7rk

is irreducible and in the discrete series for G whenever k ~ 2d + 1, and 7r2d is an irreducible representation in the limit discrete series for G (with singular co-root a:~ non-compact).

Exceptional Groups of Real Rank =4 297

4. Harish-Chandra modules and unitarizability

We recall that if 11' is an admissible representation of G on a Frechet space, the subspace of K-finite vectors is dense and affords an admissible (g, K)­module u. The representation 11' is topologically irreducible iff the module U is algebraically irreducible.

Conversely, any irreducible (g, K)-module U which admits an invariant, positive-definite inner product arises as the space of K-finite vectors in a unique irreducible unitary representation of G (cf. [H-C)). In this case, we say U is unitarizable.

Let Uk be the (g, K)-module associated to the smooth Frechet repre­sentation 11'k constructed in Proposition 3.1, for k ;:;: 2. Then 11'k is the "maximal globalization" of Uk in the sense of Schmid [S-W2; §3]. Let uk be the (g, K)-submodule associated to the unique irreducible submodule 11'1., of 11'k. It follows that uk is the unique irreducible (g, K)-submodule of Uk, and that uk is generated by the g-translates of the "minimal" K-type Sk-2(C2) ®c. The irreducible g-module uk is self-dual, by a pairing which is (-I)k-symmetric.

Our main result is the following.

Theorem 4.1. 1) The (g, K)-module Uk is irreducible and unitarizable provided k ;:;: d. 2) The irreducible (g, K)-submodule uk is unitarizable for the three further

values:

{ d -1 = 3f +3

k = 2f +2

f+2

At these values, we have the following K = SU2 X Mo decompositions, which are multiplicity-free.

U _. u' x -. 3/+3 =

=

3) Let 11'x, 11'y, and 11'z be the unitary representations of G associated to

298 B.H. Gross and N.R. Wallach

Ux, Uy, and uz. These have Gelfand-Kirillov dimensions given by:

dim(7rx) =

dim(7ry)

dim(7rz)

2d 6f+8

i(d - 1) + 1= 5f + 6

d+I 3f+5

and correspond to the nilpotent Kc-orbits pI x X, pI X Y, and pI x Z in P(lJ). The representation 7rz is the ladder representation ofG (cf. [Vj).

To prove Theorem 4.1, we first reconstruct Uk and uk via cohomological induction. Recall the O-stable parabolic subalgebra q = (EEl U, where ( = Cf3v EEl m ~ t and Uc = un t = g{3. A character A : L --+ C· gives a I-dimensional representation CA of q, annihilated by u. We view CA as a representation of the universal enveloping algebra U(q), and define

N(A) = U(g) ®U(q) CA.

Then N(A) is an admissible (g,L)-module, which has a unique irreducible quotient N(A)' generated by the image of CA. The map N ..,.. N[K) tak­ing K-finite vectors is a left-exact functor from the category of admissible (g, L )-modules to the category of admissible (g, K)-modules. We let (r~y be the right derived functors of (r~')O(N) = N[K), which were introduced by Zuckerman.

Proposition 4.2. Assume that k ~ 2 and A = -~f3. Then (rf)i(N(A» = 0 for all i =1= 1 = dim uc • The conjugate dual of the first derived functor (rf)l(N(A»V is isomorphic to the (g , K)-module Uk,

and the first derived functor (rf)l(N(A),) is irreducible, self-dual, and isomorphic to the (g, K) module uk'

The isomorphism Uk ~ (rf)l(N(A»V follows from the general com­parison theorems of Schmid and Wolf [SW2; §3], and the duality theorem for Zuckerman functors [EPWW; §6].

Proposition 4.2 reduces the question of the irreducibility of Uk to the irreducibility of the generalized Verma module N(A), and the unitariz­ability of uk to the unitarizability of the derived functor (rf)l(N'(A» for A = -~f3. Both of these questions are related to eigenvalues for the Casimir operator.

We say a weight e for L = U1 X Mo is dominant if e = -~f3 + eo with n ~ 0 and eo a dominant weight for Mo with respect to ~c' We say e occurs in an L-module W if Homd7r(e)' W) =1= O.

Proposition 4.3. The (g, L )-module N(A) is reducible if and only if

Exceptional Groups of Real Rank =4 299

there is a dominant weight e =1= A for L which occurs in N(A)U such that

lie + pll2 = IIA+pIl2.

The submodule N(A)U is contained in the L-module

N(A)Ue ~ S·(u;) ® CA

= EI1(n + k)( -~) ® sn(v). n;::O

We recall that there is a natural Hermitian form on N(A), which is non­degenerate on the quotient N(A)' [Sh]. When this form is positive-definite on N(A)'Uc, the derived functor representation (rf)1(N(A)') ~ uk is uni­tarizable [EPWW; Prop. 6.9].

Proposition 4.4 The Hermitian form on N(A)' is positive-definite on N(A)'Uc if and only if lie' + pll2 > IIA + pll2 for all dominant weights e =1= A for L which occur in N(A)'uc.

When k 2: d, it is easy to check that all dominant weights e =1= A for L which occur in N(A)Uc satisfy lie + pll2 > IIA + p1l2. Hence N(A) = N(A)' by Proposition 4.3, so Uk = Uk is irreducible. It is also unitarizable by Proposition 4.4.

When k = d - 1 we find a weight e = (4 + k)( -~) at level n = 4 in N(A)Uc with lie + pll2 = IIA + p1l2. In fact, 7r(e) lies in N(A)U and gives an exact sequence of (g, L )-modules

0----+ N(e) II

N«d + 3)(-~»

----+ N(A)

II N«d-l)(-~»

----+ N ( A)' ----+0

with N(e) and N(A)' irreducible. One checks that the condition in Propo­sition 4.5 holds for the remaining weights e' in N(A)'Uc, so u~_l is unit a­rizable.

Similarly, when k = 2f+2, f+2 we find the weights e = (3+k)( -~)+A, (2+k)(-~)+JL of levels n = 3,2 respectively in N(A)Uc satisfy lIe+pll2 = IIA + p1l2. Again, these weight spaces are annihilated by u, so give us reducibility in N(A), and Proposition 4.4 holds for the remaining weights e in N(A)'uc. This completes the sketch of the proof of Theorem 4.1.

Note. One can ask for an explicit Hilbert space representation with Harish­Chandra module uk. Perhaps the methods of Sahi [Sa] can be extended to this case, as the parabolic subgroup Q of G is defined over R and gives an analog of the Shilov boundary for D.

300 B.H. Gross and N.R. Wallach

5. Triality

Kostant has suggested that the complex Lie algebra 9 of type D 4 , together with its group 83 of outer automorphisms, should be considered excep­tional. This fits nicely into the above framework for constructing unitary representations.

The unique real form of rank 4 is the split Lie algebra 90 = 80(4,4). Its real root system has type D4; however, it could be viewed as root system of type F4 , where the 24 short roots have eigenspaces 90 of dimension 1 = O! The group G is Spin(4,4) ><183 , where Spin(4,4) is the topological double cover ofSpin(4,4)(JR). We have maximal compact K ~ 8U2 X (8U2 )3 ><183 = 8U2 X Mo, with Mo = (8U2 )3 ><183 , The representation p of K in 9 = t E9 P is isomorphic to ((:2 ® V, where

is the irreducible 8-dimensional symplectic representation of M = 8£2(C)3 ><I 83. Thus 2d = 8, so d = 4 = 31 + 4 as expected.

There are again precisely four orbits of M on JP(V), whose closures give a chain:

JP(V) = JP7

I X

I Y

I z ~ (JP1)3

The subvariety X is a quartic hypersurface, singular along Y, which has dimension = 4. However, Y is not irreducible as in the previous cases. It is the join of the closures of three 8£2(C)3 orbits, each isomorphic to JPl x p3, along their intersection Z (which is the singular locus of Y).

The coordinate algebras of JP(V), X, Y, and Z are given by Proposition 2.1, once one accepts the convention that the irreducible representation 1T(a,x + bJ.t + cv) for M has highest weight = (a + 2b + 2c, a + 2c, a) for the maximal torus of 8L2«((:)3. This representation is irreducible when restricted to 8£2«((:)3 when b = c = 0; otherwise it has 3 or 6 irreducible factors. For example, 1T(J.t) is the irreducible adjoint representation of di­mension = 9 for M, which has 3 irreducible factors when restricted to 8L2(C)3.

Proposition 3.1 is unchanged; the only modification necessary in Theo­rem 4.1 is that, since Y is not irreducible, one only obtains two small unitary

Exceptional Groups of Real Rank =4 301

representations 7r X and 7r z for G. These have Gelfand-Kirillov dimensions 8 and 5, and their Harish-Chandra modules are u; and u~, respectively. The ladder representation 7r Z of G was discovered by Kostant [K], as a submodule of a degenerate non-unitary principal series representation.

Note 5.1 The infinitesimal character A of Z(g) for the ladder representa­tion trz of the five groups G considered here is given by the W -orbit of the character A = p - ( L¥){3 of~. We have the following table:

g A of 7rz

D4 p-{3

F4 p- ~{3

E6 p- 2{3

E7 P - 3{3

Es p - 5{3.

These values for A agree with Joseph's central characters for the maximal primitive ideal in U(g) [J; Table 6.3], after some conjugation by the Weyl group.

6. Restriction

By Proposition 3.1, the representations trk of G, for all k ~ 2, are admissible when restricted to the subgroup SU2 of K = SU2 X Mo. This clearly remains true for any irreducible subquotient of 7rk, like the Frechet models 7r~ of the unitary representations 7r x, 7ry, and 7r z.

Let H c G be a closed subgroup, which contains the subgroup SU2 c K. If 7r is any unitary, SU2-admissible representation of G, it follows easily that the restriction of 7r to H decomposes discretely:

Here the Ti are irreducible unitary representations of H, the multiplicity mi of each Ti in 7r is finite, and EB is a Hilbert space direct sum.

Here we consider some restrictions of the ladder representation 7r = 7r z of the linear group G of type ES,4' The maximal compact subgroup K = SU2 X E7 / A/-L2 acts as

-ResK(7r) = EBss+n(C2 ) ® 7r(nA), n~O

302 B.H. Gross and N.R. Wallach

by Theorem 4.1 and Proposition 2.2, where 1I"(A} is the 56-dimensional, minuscule representation of E7 •

Let H be the non-compact subgroup H = SU2 x E7,4/ !If.J,2; this arises as the centralizer of an involution in G, and contains the distinguished subgroup SU2 as a factor of the maximal compact subgroup K' = SU2 x Spin12/ !If.J,2 of G' = E 7,4. We find a "Howe duality"

---ResH(1I"} = ffisn(c2 } ® Tn, n;::::O

where each Tn is an irreducible unitary representation of G' = E7,4 associ­ated to the nilpotent Kc-orbit pI x Y'. We have TO = 1I"y' for G'. More generally,

ResK'(Tn} = 9ss+n+m(C2} ® ( ffi 1I"(nwl + aw6 + bw2}) m;::::O a+2b=m

as a representation of K' = SU2 x Spin12/ !If.J,2. Next, consider the restriction when H is the subgroup Spine 4, 12) / f.J,2

of ES,4, which arises as the centralizer of the involution aiC -I} in G [B-dSJ. We again find that the restriction is multiplicity-free:

---ResH(1I") = ffi Tn·

n;::::O

Each Tn is an irreducible unitary representation of H with minimal KnH­type ss+n(C2) ® C ® 1I"(nw6}, where K n H = SU2 . SU2 . Spin12.

We may view the Harish-Chandra module u of 11" as an admissible (g,L}-module, with L = K n H. But L is a subgroup of J = Spini6 = Spin16/ f.J,2, which is a maximal compact subgroup of the split form G' = Es,s of G. Using the above decomposition of ResH(1I"), we can show that (riy2(u) is an irreducible, unitarizable (g, J)-module. Moreover, as a J­module

(ri}12(U) = ffi1l"(kA) k;::::O

where 1I"(A} is the 128-dimensional, 4-spin representation of J = Spin16/ f.J,2 on p' S;; g' = j E9 p'. This gives a construction of the ladder representation of G' = Es,s.

We have a similar construction for the ladder representations of the split groups G' = E7 ,7 and E6,6. In the first case one takes H = SU2,6/f.J,4,

Exceptional Groups 0/ Real Rank =4 303

the centralizer of the involution o:~(-I) in G = E7,4, and J = SUS/J.L4 the maximal compact subgroup of G' = E7,7. In the second case one takes H = SPl,3/ J.L2 in G = E6,4, and J = Sp4/ J.L2 the maximal compact subgroup of G' = E6,6. One finds that (ri)6(u) is the ladder representation for E7,7, and that (rfY(u) is the ladder representation for E6,6.

Finally, we can consider the restriction of 7r to the symmetric subgroups H = SL2 x E7,3/ D.J.L2 and H = Spini6/ J.L2. These do not contain the distinguished subgroup SU2, so the restriction is not discrete, but we hope to return to this problem in the future.

References

[B-dS] A. Borel and J. deSiebenthal, Les sous-groupes fermes de rang max­imum des groupes de Lie clos, Comm. Math. Helv. 23 (1949), 200-221.

[B] N. Bourbaki, Groupes et algebres de Lie, Chapter VI, Hermann (1968).

[Br] M. Brion, Invariants d'un sous-group unipotent maximal d'un groupe semi-simple, Ann. Inst. Fourier 33 (1983), 1-27.

[B-K] Brylinski, R. and Kostant, B. Minimal representations of Ea, E7 , and Es and the generalized Capelli identity, Proc. Nat. A cad. Sciences (1994), 2469-2472.

[D] Donagi, R. On the geometry of Grassmannians, Duke Math. J.44 (1977), 795-837.

[EPWW] T.J. Enright, R. Parthasarathy, N.R. Wallach, and J.A. Wolf, Uni­tary derived functor modules with small spectrum, Acta Mathemat­ica 154 (1985), 105-136.

[Ha] S.J. Harris, Some irreducible representations of exceptional algebraic groups, Amer. J. Math. 93 (1971), 75-106.

[H-C] Harish-Chandra, Representations of semi-simple Lie groups I, II, Trans. AMS 75 (1953), 76 (1954), 185-243, 26-65.

[He] S. Helgason, Differential geometry, Lie groups, and symmetric spaces, Academic Press (1978).

[I] J. Igusa, A classification of spinors up to dimension twelve, Amer. J. Math. 92 (1970), 997-1028.

[J] A. Joseph, The minimal orbit in a simple Lie algebra and its asso­ciated maximal ideal, Ann. Sci. E.N.S. 9 (1976), 1-30.

[K-S] D. Kazhdan and G. Savin, The smallest representation of simply laced groups, Israel Math. Con!. Proceedings 2 (1990), 209-223.

[K] B. Kostant, The principle of triality and a distinguished unitary representation of SO( 4,4). In: Differential geometric methods in

304 B.B. Gross and N.R. Wallach

theoretical physics, Kluwer (1988), 65-109. [K-R] B. Kostant S. and Rallis, Orbits and representations associated with

symmetric spaces, Amer. J. Math. 93 (1971), 753-809. [0-V] A.L. Onishchik E.B. and Vinberg, Lie groups and algebraic groups,

Springer Soviet Math., (1988). [Sa] S. Sahi, Explicit Hilbert spaces for certain unipotent representations,

Inventiones Math. 110 (1992), 409-418. [Se] c. Segre, Sui complessi lineari di piani nello spazio a cinque dimen­

sioni, Annali di Math. Pum ed Applicata 27 (1918), 75. [S-K] M. Sato and T. Kimura, A classification of irreducible prehomoge­

neous vector spaces and their relative invariants, Nagoya Math. J. 65 (1977), 1-155.

[Sw] G.W. Schwarz, Representations of simple Lie groups with regular rings of invariants, Invent. Math. 49 (1978), 167-191.

[S] W. Schmid, Homogeneous complex manifolds and representations of semisimple Lie groups; Thesis, Berkeley, 1967. In: Representa­tion theory and harmonic analysis on semisimple Lie groups, AMS Surveys and Monographs 31 (1989), 223-286.

[S-W] W. Schmid J.A. and Wolf, A vanishing theorem for open orbits on complex flag manifolds, Proc. AMS 92 (1984), 461-464.

[S-W2] W. Schmid and J. Wolf, Geometric quantization and derived functor modules for semisimple Lie groups, J. Funct. Anal. 90 (1990),48-112.

[SH] N.N. Shapovalov, On a bilinear form on the universal enveloping algebra of a complex semi-simple Lie algebra, Funct. Anal. Appl. 6 (1972), 307-312.

[V] D. Vogan, Singular unitary representations. In: Non commuta­tive harmonic analysis and Lie groups, Springer Lecture Notes 880 (1981), 506-535.

[W] N. Wallach, Real reductive groups I, II. Academic Press 1988, 1992. [Wo] J.A. Wolf, Complex homogeneous contact manifolds and quater­

nionic symmetric spaces, J. Math. Mech. 14 (1965), 1033-1047.

B.H. Gross, Department of Mathematics, Harvard University, Cambridge, MA 02138

N.R. Wallach, Department of Mathematics, University of California at San Diego, La Jolla, CA 92093

Received September 1, 1993

Reduced Phase Spaces and Riemann-Roch

Victor Guillemin*

Introduction

We will be concerned in this article with the conjecture

Q(Ma) = Q(M)a (0.1)

where G is a compact Lie group, M a pre-quantizable Hamiltonian G­space, Ma the associated reduced space, Q the "quantization functor" and Q(M)a the G-fixed vectors in Q(M). It is customary to think of Q(M) and Q(Ma) as Hilbert spaces; however, in this article they will be defined to be elements of the K-groups, Ka(pt) and K(pt). More explicitly, Q(M) will be defined to be 1I"!LM where LM is the pre-quantum line bundle on M, 11" is the map, M ...... pt, and 11"! is the "lower-shriek" operation in the equivariant K-theory. (see §6. Q(Ma) will be defined similarly.)

In principle, (0.1) ought to be provable in the cases in which it is true by K-theoretical techniques; however, in this article our approach to (0.1) will be via localization theorems in equivariant De Rham theory. Using some recent results of Jeffrey and Kirwan, we will compute the Riemann­Roch number of M a , which is the dimension of the left hand side1 and we will compare this with the dimension of the right hand side, which can easily be computed by the Atiyah-Bott fixed point formula, and this will give us a combinatorial identity2 which is known to be true in a lot of cases.

To make the details of this proof comprehensible (to ourselves as well as our readers) we will assume for simplicity that M is compact, G abelian, and the fixed point set of G finite. Moreover, since G is abelian we will

* Supported by NSF grant DMS 890771. 1 In this paper we will assume that Ma is a manifold (c.f. sopra) and

hence that its Riemann-Roch number is given by the Riemann-Roch for­mula of Hirzebruch-Atiyah-Singer. In general, however, Ma is an orbifold, so its Riemann-Roch number is given by the Kawaski Riemann-Roch for­mula. (The orbifold case will be discussed in the second installment of these notes.) 2 Namely, the combinatorial Riemann-Roch formula of Khovanskii (cf. [KPj and [KKJ).

306 v. Guillemin

consider in place of (0.1) the slightly more general conjecture

(0.2)

where q is in the weight lattice of G, Mq is the reduced space associated with q and Q(M)q is the subspace of Q(M) spanned by weight vectors of weight q. To insure that Mq is non-singular we will assume that for certain fixed points, p, the linear isotropy representation of G on the tangent space at p is quasi-free and that q is a regular value of the moment map associated with this representation. The regular values of this moment map are a disjoint finite union of convex polyhedral cones, so this amounts to asserting that, for each p, q lies in one of these cones. We will also assume that for each p, q lies in the same open cone as the shifted weight

q+ ~ (Eok',p - EOk,P)

where Ok,p, k = 1, ... , d, are the weights of the linear isotropy represen­tation and ok',p their "polarizations".3 Some of these assumptions will hopefully turn out to be unnecessary; however, even as things stand, the result we have obtained here is quite a bit simpler than our earlier version of (0.2) in [GS,l] (and the proof is much more elementary).

These notes consist of six sections. The first two sections are a review of material from a recent paper of Jeffrey and Kirwan, "Localization for non-abelian group actions." (Despite its title, the contents of this paper are of considerable interest for abelian groups as well, and we will mainly be interested here in the abelian version of their results.) In §3 we will show that, morally speaking, (0.2) is a special case of the Jeffrey-Kirwan localization theorem. Unfortunately we haven't been able to make the proof we sketch of (0.2) in §3 completely rigorous; however, we have been able to salvage from it an argument which shows that (0.2) is true for actions of G on compact manifolds providing it's true for linear actions of G. We describe this argument in §4 and in §5 we show that, in the linear case, (0.1) reduces to a combinatorial theorem about counting lattice points in convex polytopes. Finally, in §6 we state our main theorem and make a few concluding remarks.

We would like to thank Raoul Bott and Lisa Jeffrey for helpful com­ments. We would also like to thank Hans Duistermaat for suggesting to us

3 For the definition of oi:,p see §2. These conditions on the shifted weights are related to the fact that Q(M) is a "lower shriek" rather than a simple "push-forward."

Reduced phase spaces and Riemann-Roch 307

that our proof of (0.2) in the Kaehler case in [GSIJ might be simplified con­siderably by making use of localization techniques in equivariant DeRham theory.

1. The localization theorem in equivariant cohomology

Let G be an n-dimensional torus, and let 9 be the Lie algebra of G. Let M be a 2d-dimensional manifold and

r: G -+ Diff( M) (1.1)

an action of G on M. We briefly recall how the equivariant cohomology groups, Ha(M), are defined: Fix a basis eb e2, ... , en of g. For each ek

one gets by (1.1) a vector field on M and hence an anti-derivation of the DeRham complex of degree -1:

Le ,,: n(M) -+ n(M) (1.2)

Now define an anti-derivation operator, D, on the tensor product

(1.3)

by setting

D(1 ®xl") = 0 (1.4)

n

D(w® 1) = dw® 1- HLLe"W®Xk (1.5) k=l

for (,; E n(M).4 It is easy to check that D2 = O. Moreover, if one gives the ring (1.3) the grading

deg(1 ® xl") = 2/J-L/ ,

4 This definition of D is somewhat non-standard but is consonant with PKJ. (See PKJ, footnote 8.) One gets back the standard definition by making the change of coordinates, Xk -+ y'=Ixk, in the polynomial ring C[Xb ... ,XnJ.

308 V. Guillemin

and

deg(w ® 1) = deg(w) ,

D is an anti-derivation of degree 1. The cohomology ring H'G(M) is now defined to be the cohomology ring of the complex (1.3) equipped with this differential. For future reference we note that

(1.6)

and, if the action (1.1) is locally free,

H'G(M) = H*(MIG, C) , (1. 7)

the right hand side being the usual DeRham cohomology of the orbifold, MIG. (The assertion (1.6) is obvious. For (1.7) see [AB,l].)

If M is compact and oriented, which is what we will assume from now on, the map, 71": M -+ pt, gives rise to a push-forward operation in cohomology

71"# : HG(M) -+ HG(pt) . (1.8)

This operation lowers degree by -2d; and, at the level of forms, maps the closed equivariant form,

(1.9)

onto the element

(1.10)

of C[Xl,' .. ,xn ]. The localization theorem (lAB,l], [BV]) is a recipe for evaluating this integral in terms of data on the fixed point set, M G , of G. This recipe is easiest to state when MG is finite, which is the only case we will consider here. In this case the theorem says that

(1.11)

where the "residue," Tp«(3), of the form (1.9) at a fixed point, p, is given by

(1.12)

Reduced phase spaces and Riemann-Roch 309

In this expression, Ok,p, k = 1, ... ,d, are the weights of the linear isotropy representation of G on the tangent space at p and tp is the inclusion of p

into M. Thus, by (1.9),

wZ being the degree zero component of the form, Ww One puzzling feature of this formula is that these terms are rational functions of x whereas the left hand side is a polynomial. (Thus a lot of cancellations are occurring on the right hand side.)

The localization theorem of Jeffrey and Kirwan is a refinement of (1.11) for the Hamiltonian G-actions: Assume that 1/ is a G-invariant symplectic form on M and (1.1) a Hamiltonian action. Then, by definition, there is a moment map

¢:M -+ g*

with the property

(1.13)

Let q E g* be a regular value of ¢ and let Zq be the level set, ¢-l(q). By (1.13) the action of G on Zq is locally free; and, in particular, the reduced phase space

(1.14)

is an orbifold. Let t: Zq -+ M be the inclusion map. By (1.7)

and, combining this with the restriction map

one gets a map

K,q: Ha(M) -+ H*(Mq, C) . (1.15)

Let l/q be the reduced symplectic form on M q • The map

310 v. Guillemin

induces a map on cohomology

H*(Mq, C) - C (1.16)

which, composed with (1.15), gives one a map:

Tq: Ha(M) - C . (1.17)

Given aD-closed equivariant form, f3, the result we are heading for is a residue formula for Tq[f31 similar to (1.11). This we will derive from (1.11) itself by the following stratagem: Let

(1.18)

be the "equivariant" symplectic form. This form is D-closed, by (1.13); hence the product

f3 /\ exp v# = (f3 /\ exp v)ei(cP.x ) (1.19)

is also aD-closed equivariant form. 5 Now factor the map, 1T: M - pt into the composition of ¢: M - g* and 1To: g* - pt, and express the push­forward

(1.20)

as the push-forward by (1To)* of

(1.21)

Let's analyze this last expression more carefully. By (1.9) it can be written as the sum

(1.22)

where the es are the coordinates on g* dual to Xl. ... , x n .6

5 This product is not, strictly speaking, an element of the complex (1.3); however this will not pose serious difficulties in the argument we are about to sketch, since one can expand expi(¢, x) into its Taylor series and deal separately with each term. However, we would not be able to deal so cava­lierly with this exponential factor if f3 were also a transcendental function of x. (This is a situation we will encounter in §3.) 6 An important parenthetical remark: The expressions, ¢* (w IJ- /\ exp iv)

are no longer DeRham forms. They are currents!

Reduced phase spaces and Riemann-Roch 311

Letting

one can rewrite (1.22) as

and the push-forward of this by (1I"0)x is the same as the push-forward of

(1.23)

Let h(e)de be the component of degree n of this current. Then the push-forward of (1.23) by (11"0). is just the integral

which, up to a factor of (211")n, is the inverse Fourier transform of h(e). Thus we have proved:

Theorem 1.1. Let

f(x) = 11".({3/\ expiv#) . (1.24)

Then j(e) (211")-nh(e) where h(e)de is the degree n-component of the compactly supported current

I)-D)I'4>.(wl' /\expv) . (1.25)

Therefore, by (1.11) we can write f(x) as a sum of residues, compute the Fourier transform of each summand and compare with (1.25). We will carry out the details in the next section. First, however, we will show that the current (1.25) is smooth at regular values, q, of 4> and that

(1.26)

Thus, as a corollary of theorem 1.1, we will get the following result (which is the localization theorem of Jeffrey-Kirwan in the abelian case):

312 v. Guillemin

Theorem 1.2. If q is a regular value of the moment map,

(1.27)

f being the function {1.24}.

The proof of (1.26) will be by means of the following "canonical model" for M near the level set, Zq. Let () be a G-invariant g-valued one-form on Zq with the property

(For the existence of a "connection form" of this type, see [BV].) Let p be the composite map

and let IIq be, as above, the reduced symplectic form on M q • Then the two-form

P*lIq + d(f., () (1.28)

is a symplectic form on Zq x g* (at least in a neighborhood of Zq x {O}). Moreover, if one lets G act on this product by acting trivially on the second factor, this action is Hamiltonian and the resulting moment map is the map,

(z, f.) E Zq x g* ~ f. + q . (1.29)

Finally since the action of G on Zq x g* is locally free one has an isomorphism

Ha(Zq x g*) ~ H*(Mq x g*,C), (1.30)

and, since Mq is a deformation retract of Mq x g*, an isomorphism

H*(Mq x g*,C) ~ H*(Mq,C). (1.31)

Thus every equivariant D-closed form on Zq x g* is D-cohomologous to a form of the type p*w, where w is a closed form on M q •

For the following see [GS,2] or [Mar].

Lemma. The product, Zq x g*, equipped with the form {1.28}, is a local canonical model for (M, II) in a neighborhood of Zq.

Reduced phase spaces and Riemann-Roch 313

Thus to compute (1.25) in a neighborhood of ~ = q, we can take M to be Zq x g* and take /3 to be an ordinary d-closed DeRham form of the form, p*w, where w is a closed form on M q • With these normalizations we will prove:

Lemma 1.3. The expression {1.25} can be written in a neighborhood of ~ = q as a sum

the hI(~) 's being polynomials in~. Moreover, if h(~) is the coefficient of the summand of degree n:

(1.32)

Proof. Let 0 = E eiOi. With /3 = p*w and v equal to (1.28), (1.25) is the push- forward by the moment map, (1.29), of the form

In view of the shift by q in (1.29) this is equal, on the q-Ievel set of the moment map, to

p*(w /\ exp Vq)Ol /\ ... /\ On /\ d6 /\ ... /\ ~n + ...

Therefore, its push-forward is equal, at ~ = q, to

(The factor (27r)n comes from the integration of (h /\ ... /\ On over G.)

2. The Jeffrey-Kirwan localization theorem

We still have to compute the Fourier transform of the function

(2.1)

314 v. Guillemin

where (3 and y# are given by

(2.2)

and

y + R(¢, x) . (2.3)

The computation of f(x) itself is easy: By (1.11), f = E f p , p E MG, where

On the other hand f (x) is also equal to the integral

(2.5)

and from this expression for f it is clear that f is a function of tempered growth, and hence that its Fourier transform is well-defined as a tempered distribution. However, it is not so easy to define the Fourier transforms of the individual fp's. For instance let ha, for a E g*, be the distribution

(2.6)

where cp is in the space of test functions, C8"(g*). Then Daha = -A8o; so the inverse Fourier transform of ha satisfies

a(x)h~(x) = -H(21r)-1 ,

i.e. on the set a(x) =1= 0:

(2.7)

Let t;(3(D) be the constant coefficient differential operator

(21r)n Lw~(p)DI' . (2.8) I'

Then by (2.7) the convolution product

(2.9)

Reduced phase spaces and Riemann-Roch 315

is the Fourier transform of a tempered distribution which is equal to fp on the set where the ai,p's are non-zero. This, however, is not the only distribution with this property. For instance, the inverse Fourier trans­form of -h_a also satisfies (2.7); so if we replace some of the factors, hOI, (a = ak,p) in the product (2.9) by -h_a we get another distribution with the same property.7 There is, however, a consistent way of making these choices of sign: Consider the subset of g consisting of the union of the N d hyperplanes

(2.10)

where N is the number of fixed points. The complement of this set has several connected components which we will call Weyl chambers. Now fix a Weyl chamber, W, and for each p define the polarized weights at p to be

(2.11)

where fi = +1 if ai,p(v} > 0 for all v E Wand fi = -1 if ai,p(v) < 0 for all v E W. Also let

( -l)P = 11 fi . (2.12)

With these normalizations we will prove:

}(I;,) = I)-l)Pt;.B(D} (8q,(p) * h~l,P * ... * h~d'P) (2.13) P

Proof. Let it be the inverse Fourier transform of the right hand side. By construction f = it on the complement of the set (2.10); therefore, there exist positive integers, Ni,p, such that

(11 ai,p(x)Ni'P) (f - it) = 0

and hence

(11 D::::) (i - it) = 0 . (2.14)

However, } is compactly supported by (1.27) and iI, which is the right hand side of (2.13), is supported on a half-space of the form

(I;, v) ~ -Const

7 More generally we can replace hOI by clha + C2ha + 9 where Cl - C2 = 1 and Dag = o.

316 V. Guillemin

where v is any element of W. Thus 1 - 11 is also supported on such a half-space. This, plus the fact that 1 - A satisfies a differential equation of the form (2.14), implies that A = f.8

At regular values of the moment map, rq{(3) is equal, by (1.27), to the value at e = q of the right hand side of (2.13). However, the fact that 1{e) is smooth at q does not imply that the p-th term on the right hand side of (2.13) is smooth at q for all p. Therefore, it may not be possible to compute rq{(3) by (2.13) for all regular values of the moment map. We will show, however, that one can do this for an open dense subset of the set of regular values. To simplify notation let J.L = cf>{p) and ai = a~p' Let Rt be the positive orthant in Rd {i.e. (XI. ••• , Xd) E Rt iff Xi ;:::: 0 for all i), and consider the map

Since ai{v) > 0 for all v E W this map is proper, and its image is contained in W* + J.L (where W* is the dual cone to W.) For the following elementary results, see [OLS], section 3.

Lemma 2.1. Let m be Lebesgue measure on Rt. Then

(2.15)

This lemma has the following corollary: Let 6{e) be the convex poly­tope consisting of all points, (Xl, ... , Xd) E Rt satisfying

(2.16)

Then:

Proposition 2.2. If aI, ... ,ad span g* the convolution 81' * hal * ... * had

is continuous as a function of e and is equal to volume 6(e - J.L).

In fact a much stronger regularity result is true. Consider the union of

8 The theorem involved here is the following. Let VI, ••• , Vd be vectors in R n and let f be a distribution with support in a half space H. Then if DVI •.. DVdf = 0 and none of the Vi'S are tangent to the boundary of H, f is identically zero. Proof: Induction on d (the case d = 1 being obvious).

Reduced phase spaces and Riemann-Roch 317

all sets of the form

(2.17)

where k < n.

Proposition 2.3. The restriction of the function, volume(6.(~»), to any connected component of the complement of (2.17) is a polynomial in ~ of degree d - n.

Putting the p-dependence back into the picture, consider the union over p E MG of the sets

k

:~:::>irO:i:.,p , x E Rt , (2.18) r=l

where k < n and let 6.p(~) be the polytope consisting of all (Xl, ... ,Xd) E

Ri which satisfy

(2.19)

Then the results above imply:

Theorem 2.4. For ~ in the complement of the set (2.18)

rd(3) = ~) -l)Pt;f3(De) volume(6.P(~ - ¢(p))) (2.20) P

where t;f3(D) is the constant coefficient differential operator whose symbol is the polynomial t;f3(x).

Two comments about this formula: 1. The terms in t;f3(x) of degree greater than d - n make no contribution

to (2.20). 2. Let v E W. Then if (~,v) is less than (¢(p), v) the p-th term in (2.20)

is zero. In other words, on the right hand side, the only non-zero summands are those for which ~ - ¢(p) E W*.

318 V. Guillemin

3. Riemann-Roch

We will now assume that there exists a line bundle, LM ---+ M, and a connection, YO, on this bundle whose curvature is the symplectic form, v. We will also assume that the action of G on M lifts to an action of G on LM and that the action of G on sections of LM is given infinitesimally by the map

v E g ---+ yo tI + 211'i(q" v} .

Let q be a regular value of q" let Zq = q,-l(q) and let Mq = Zq/G. We would like there to exist a line bundle, Lq ---+ Mq and a connection, yo q, such that

and (3.1)

where 11' is the projection of Zq on Mq and t the inclusion of Zq into M. To guarantee the existence of Lq and yo q we will assume from now on that q belongs to the weight lattice in g* and that G acts freely on Zq . (See [GS,lj. The first assumption is a necessary condition for the existence of Lq and yo q; and if G acts freely on Zq, it is sufficient as well.)

Now equip T M with a G-invariant complex structure compatible with v and TMq with a complex structure compatible with vq. These complex structures are unique up to isotopy; and, in particular, the Todd class, r(TMq), is independent of the choice of the complex structure on TMq. The Riemann-Roch number ofLq is, by definition:

(3.2)

where c(Lq) is the Chern class of Lq , and since

by (3.1), the expression (3.2) can also be written:

(3.3)

In particular suppose there exists an equivariant cohomology class, (3,

Reduced phase spaces and Riemann-Roch 319

with the property

(3.4)

where "-q is the map (1.15). Then by (1.17)

R.R.(Lq) = Tq(f3) (3.5)

and hence the Riemann-Roch number of Lq can be computed from data at the fixed points by (2.20). There is, fortunately, a canonical choice of a class, f3, with the property (3.4). To see this we will first prove:

Lemma 3.1: There is a G-equivariant isomorphism of complex vector bundles

(3.6)

Proof. By assumption G acts freely on Zq, so Zq can be regarded as a principal G-bundle with Mq as base. A connection on this bundle gives a splitting

Moreover, since Zq = lfJ-l(q) the normal bundle of Zq is trivial (with fiber, g*) and hence

t*TM=TZEBg* .

Finally, from the Hermitian structure on t*TM one gets an identifica-tion

T Zq EB g* = T Zq EB Hg .

Q.E.D.

Next note that since G is abelian, the adjoint action of G on Qc is trivial. Hence, on the left hand side of (3.6), the second summand is not only trivial as a vector bundle but is even trivial as an equivariant vector bundle. Therefore,

(3.7)

320 v. Guillemin

where r(TM) is the equivariant Todd class of TM, and hence (3.4) is satisfied if we take f3 = r(T M). Thus by (3.5)

R.R.(Lq) = Tq(r(TM»

so from (1.27) one gets

(3.8)

where

J(x) = 1M (r(TM) exp v#) . (3.9)

By the (1.11) localization theorem the right hand side of (3.9) is the sum, over p E MG, of:

(3.10)

Moreover,

and the equivariant Todd class of the representation of G on T pM is

so, substituting this into the expression above, one ends up with the formula

(3.11)

We now have to compute the Fourier transform, j(f;.), and evaluate it at f;. = q. This we will do as follows: From the almost-complex structure on M one gets an almost-Dolbeault complex9

L "8 nO 1 "8 M~LM®H' ~ ... (3.12)

9 This is an "almost" -complex in the sense that li is a differential operator of order zero.

Reduced phase spaces and Riemann-Roch 321

Moreover, from the Hermitian structures on LM and on the nO,i,s one gets adjoints, a*, of the operators, a. We define the spin-C Dirac operator, /Jc, to be the restriction of a + a* to the even part of the complex above. (Thus it maps the even part of this complex on to the odd part.) Let Q( M) be the virtual Hilbert space

kernel(/Jc) - cokernel(/Jc) . (3.13)

There is a natural representation of G on Q(M) for which one has the following result:

Theorem 3.2. The character of the representation of G on Q(M) is the function (3.11}.lO

Proof. This is just the Atiyah-Bott fixed point formula for /Jc. See [AB, 2], §4.

As a corollary of this theorem one gets for the Fourier transform:

(3.14)

where the sum is over the weight lattice and #(-Y, Q(M» is the multiplicity with which the weight 'Y occurs in the representation of G on Q(M). If we evaluate this at e = q and ignore the nonsensical factor, Dq(q), (or rather renormalize it by setting it equal to one) we end up with

R.R.(Lq ) = # (q, Q(M» (3.15)

which is what we would expect from (0.1). (In the next section we will show how to get (3.15) without this renormalization.)

4. Some computations with virtual Chern classes

Let X be a compact manifold and E a complex vector bundle over X of dimension d. We will define the virtual Chern classes of E to be the roots

i = 1, .. . ,d (4.1)

of the equation

(4.2)

lO Since the (}:k,p'S and ¢(p)'s are in the weight lattice, (3.11) can be viewed as a function on G.

322 v. Guillemin

where ci(E) E H2i(X, C) is the standard i-th Chern class of EY If, in addition, X is a G-space and E a G-bundle, we will define the equivariant virtual Chern classes to be the corresponding elements of H't;(X).

In particular let Vi, i = 1, ... , d, be the virtual Chern classes of the bundle

TMq E9gc,

and let vt, i = 1, ... , d, be the equivariant virtual Chern classes of the bundle, TM. By (3.6)

and hence

By (1.27) the left hand side of this equation is equal to the value at , = q of the Fourier transform of the function

On the other hand this function itself is equal to the sum, over p E M G ,

of the expressions

However ··v# = ei(q,(p),x} and by functoriality· , ~p " •

since the equivariant virtual Chern classes of the representation of G on Tp are iO:1,p(X), •.. , iO:d,p(X). Thus, f(x) is equal to

i-d L ei(q,(p)+ L: tkuk,p,X} (II O:k,P(X») -1

p

11 Just like certain elementary particles, these classes are virtual in the sense that they can only exist in symmetric combinations such as 1/1 + ... + Vd, V? + ... + VJ, etc.

Reduced phase spaces and Riemann-Roch 323

and hence by (2.20), J(f.) is equal to the sum over p E MG:

L ( -1)P volume Lif (f. - 4>(P))

where Lif is the convex polytope:

{ (Xl, ... ,Xd) E R! j L tkQk,p + XkQk',p = f.} .

N.B.: By (2.11) Qk,p = fk,pQk,p (where fk,p = 1 if Qk,p(V) > 0 for all v in the Weyl chamber, Wj and fk,p = -1 if Qk,p(V) < 0 for all v in W), so the polytope above is also defined by the equation:

(4.4)

Since the right hand side of (4.3) is equal to j(q), we end up with the following identity:

This is of course just a virtual identity, however, one can get from it interesting expressions for the "twisted" Chern numbers of Mq as follows: Let

be a polynomial which is invariant with respect to permutations of x}, ••• ,

Xd, and let

Applying this constant coefficient differential operator to both sides of (4.5) and setting t = 0, one obtains:

fMq

exp IIq P( lIlt .. . ,Vd) = L( _1)d (p (!) volume Lif(q - 4>(p») 0 .

(4.6) In this identity the left hand side is now well-defined since P(VI, ... , lid)

is a polynomial in the usual Chern classes. As for the right hand side, it is also well-defined providing q is in the complement of the set (2.18), in which case, by proposition (2.3), the function

t ~ volume Lif( q - 4>(p»

324 v. Guillemin

is a polynomial of degree d - n for t in a neighborhood of o. The formula (4.6) even makes sense when P is a formal power series,

since the homogeneous terms in this power series of degree greater than d - n make no contribution either to the left hand side or the right hand side. In particular let

and let P be equal to

X T(X) = -1---e---'"

d

Td(Xl, . .. ,Xd) = II T(Xi) . i=l

Then the left hand side of (4.6) becomes:

and on the right hand side, one gets

2:) -1)1' (Td (!) vOlume.!if (q - ¢(P») 0

(4.7)

(4.8)

We will now transform this into a slightly different form. Note that:

and hence

(4.9)

where T-l is the operator, (T_lcp)(y) = cp(y - 1), and cP is a polynomial in y. Now let

i = 1, ... ,d (4.10)

and let A~(O be the polytope

(4.11)

Reduced phase spaces and Riemann-Roch 325

Then, by (4.10)

Moreover, if J(t) is a polynomial in t and yes) = J(t) then by (4.9):

Td (!) J(t) = Tp OTd (!) yes) (4.12)

where

2J.L = (fl,p - 1, ... , fd,p - 1)

and Tp is the operator "translation by J.L." Let Ap be the complement of the union of the sets

k

L XirQ;i';,p , x E R~ , r=l

where k < n, and recall that for the p-th term in the sum (4.8) to make sense, we have to assume that q - <p(p) is in Ap- We will now make a stronger assumption about q. Let

1 'Yp = 2 L Q;k,p

and (4.13)

w 1~ w 'Yp = 2 L...,; Q;k,p •

We will assume that both q - <p(p) and q - <p(p) + "I;' - 'Yp are in Ap and, in addition, that they are in the same connected component oj Ap. Since the connected components of Ap are convex cones, this implies that the line segment joining q - <p(p) to q - <p(p) + "I;' - 'Yp is in Ap- With these assumptions we will prove:

Lemma 4.1. The p-th term in the sum (4.8) is equal to

Td (:s) (volume ~~(q - <p(p) + "I;' - 'Yp») (4.14)

evaluated at s = o.

Proof. By Proposition 2.3, the function, volume ~~(q - <p(p» is a poly­nomial in s on an open convex set containing the origin and J.L. Hence, by

326 v. Guillemin

(4.12), the p-th term in the sum, (4.8), is equal to

T (!) (volume ~~(q - ¢(p)))

evaluated at S = pj and this, by (4.11) is equal to the expression above. As a corollary of this lemma we get the following result (which is the

main theorem of this section):

Theorem 4.2. Assume that for all p E M G , the weight vectors, q - ¢(p) and q - ¢(p) + 0;:' - op lie in a fixed connected component of Apo Then R.R.(Lq ) is equal to the expression

(4.15)

evaluated at s = o.

We will compare this with a very similar formula for the weight multi­plicity, #(q, Q(M». Let

(4.16)

(In other words, Np(q) is the number of integer lattice points in Rd which are contained in the convex polytope, ~p(q).) Then

# (q, Q(M» = ~) -1)P Np (q - ¢(p) + ')'; - ')'p) (4.17)

(See [GLSJ.)12 Hence, by theorem 4.2 the proof of the identity (3.15) reduces

12 For the proof of (4.15) see [GLSJ, §4. By the Atiyah-Bott fixed point formula, the distribution

L #(q, Q(M»oq ,

summed over the weight lattice, is equal to the Fourier transform of

The proof of (4.15) consists of showing that the Fourier transform of this

Reduced phase spaces and Riemann-Roch 327

to proving

(4.18)

for all p E Ma and all q E Ap. This identity, of course, has nothing to do with symplectic geometry. It is an analytic formula for counting the number of lattice points in a convex polytope which mayor may not be true. (At the moment there is no evidence either way.) We will show below, however, that (4.18) is a special case of (3.15): Namely (3.15) reduces to (4.18) if M is a symplectic vector space and the action of G on M is linear. We will also show that (3.15) is true in the linear case if the linear action is quasi-free.

5. The linear case

Let V be a complex d-dimensional vector space, H a positive definite Hermitan form on V and

T:G --> U(V) (5.1)

a unitary representation of G. Since T preserves the symplectic form

w(v,w) = 1m H(v,w) (5.2)

one can think of T as a Hamiltonian action of G on V. The following facts are easy to prove, but we won't take the time out to verify them here. (For the first four propositions below see [GLSj, for proposition 5.5, see [DGPj, and for propositions 5.6 and 5.7, see [Gj.)

Let el, ... , ed be an orthonormal basis of V with the property that the one-dimensional spaces spanned by the ei's are G-invariant.

Proposition 5.1. The moment map associated with T is the mapping

(5.2)

function is also equal to

'L( -l)P Np (q - ¢(p) + /';' - /'p) Dq . q,p

328 v. Guillemin

where (Z1, .. . , Zd) is the coordinate system associated with the basis above and O:i is the weight of the representation of G on the one-dimensional subspace spanned by ei.

Proposition 5.2. The action, T, is effective if and only if 0:1, ••• , O:d span the weight lattice of G in g* .

Proposition 5.3. The map (5.2) is proper if and only if there exists a vector, v E g for which O:i(V) > 0 for all i.

We will assume from now on that T is effective and T is proper. In other words we will assume that the hypotheses on the O:i'S in the two propositions above are satisfied. It is clear from the form of (5.2) that the image of ¢ is the convex cone

(5.3)

Let C be the subset of this cone consisting of all points, L XiO:i, with the property that

# {Xi, Xi =1= O} < n .

Proposition 5.4. C is the set of critical values of the moment map, ¢.

Thus its complement, A, is the set of regular values of ¢. For q E A let Zq be the level set, ¢(P) = q, and let Mq be the orbifold, Zq/G. For this orbifold to be a manifold, G has to act freely on Zq, and hence, for this orbifold to be a manifold for all q E A, T has to be a quasi-free action. 13

The following is a necessary and sufficient condition for this to be the case:

Proposition 5.5. The action, T, is quasi-free if and only if, every linearly independent set of weights, O:ip ••• , O:in , is a basis of the weight lattice in g*.

We will assume from now on that this condition holds as well. Hence, as a consequence, for all q E A, Mq is a compact symplectic manifold. We will show that this manifold has a large group of symmetries. Let Td be

13 An action of G on a set, S, is quasi-free if, for all s E S, the stabilizer group is either the identity or is of dimension greater than zero.

Reduced phase spaces and Riemann-Roch 329

the standard d-dimensional torus

8 1 X ••• X 8 1 , (d factors) ,

and let

be the unitary representation of Td on V defined by

(5.4)

The fact that G acts effectively on V means that there is a unique imbedding, G _ T d , defined by the property that the restriction of 71

to G is 7. In particular, 71 is a Hamiltonian action of Td on V which commutes with 7j and hence, on each of the reduced spaces, M q , one gets an effective Hamiltonian action of the quotient group

Therefore, Mq is a Delzant space; and, by Delzant's theorem, is deter­mined up to isomorphism by its moment poly tope. 14 We claim that this is the polytope we encountered in §4, namely:

(5.5)

Proof. By (5.4) the moment map associated with the action of Td on V is the map

and, by (5.2), ~(q) is the image of the restriction of this map to Zq. The next few results illustrate how properties of Mq can be "read off"

from this polytope. The simplest of these is the following:

14 A compact Hamiltonian T -space is Delzant if T acts effectively and the dimension of the space is twice the dimension of T. The moment polytope is the image of the moment map, and Delzant's theorem (c.f. [Del)) asserts that if two such spaces have the same moment polytope, they are T-equivariantly symplectomorphic.

330 v. Guillemin

Proposition 5.6. The symplectic volume of Mq is equal to the volume of ~(q).

Suppose now that Mq is pre-quantizable (Le. suppose q is a weight vector). Let Lq be its pre-quantum line bundle. As a corollary of proposi­tion 5.6, we will derive below a formula for R.R.(Lq ) involving ~(q). First note, however, that the representation (5.1) can be written as a direct sum:

T = Ti EB ••• EB Td , (5.6)

Ti being the representation of G on the subspace spanned by ei. Moreover, since G acts freely on Zq, Zq is a principle G-bundle with Mq as base. Hence, for each Ti, one gets an associated line bundle

Let Vi be the Chern class of this bundle.

Proposition 5.1. The cohomology class, Vq, of the reduced symplectic form on Mq is E qiVi.

For M = V, TM is the trivial bundle with fiber V, so (3.6) reduces to:

and thus by (5.6)

and hence one deduces:

Proposition 5.S. The cohomology classes, Vi, ... , Vd, are the virtual Chern classes ofTMq EB gc.

In particular, by Proposition 5.7, the Riemann-Roch number of Lq is equal to

However, by proposition 5.6 and proposition 5.7:

1 exp (E qiVi) = volume ~(q) ; Mq

(5.7)

Reduced phase spaces and Riemann-Roch 331

so by (5.7) one ends up with the following formula:

R.R.(Lq ) = Td (:q) volume ~(q) . (5.8)

Let's use this formula to check if the identity (3.15) is true when M = V. Unfortunately, since V isn't compact, there is no canonical way to define Q(V) up to isomorphism; however, a sensible choice of Q(V) is the space of holomorphic functions, f, satisfying

id J If(vWexp(-H(v,v)/2)dvav < 00 •

The elements of this space which transform according to the weight, q, are the monomials

for which

so the multiplicity with which q occurs as a weight in Q(V) is given by the partition function which we encountered in §4, namely

N(q) = {m E Z~ , Lmioi = q}

Thus, in this linear case (3.15) reduces to:

N(q) = T (:q) volume ~(q) (5.9)

which is the identity (4.18) with 0i = 0i:'p. We will now describe what is known about this identity:

1. This identity is true if the weights, 01. ... , 0d, satisfy the hypotheses of proposition 5.5 (in other words if the representation, (5.1), is quasi­free). This was proved by Khovanskii and Pukhlikov in [KPJ.

2. Khovanskii and Pukhlikov actually proved a much stronger result: Let P(x) be a polynomial in Xl. ••• , Xd. Then for weight vectors, q E A,

T (a a ) f P(x)dx = L P(m) , q } L1(q)

(5.10)

332 v. Guillemin

the sum on the right being over all lattice points, m E Zd. Their proof of this is combinatorial, and, in particular, doesn't use any simplectic geometry. (The symplectic implications of (5.10) are as yet unclear.)

3. Prior to this, Danilov had proved directly that

without showing that the right hand side of (5.9) is given by 7r(8j8q) volume ~(q). Thus, in particular, Danilov's result ([Dan]) coupled with the argument above, gives an alternative (symplectic) proof of (5.9).

4. Variants of (5.9) are known to be true when the weights, at. ... , ad,

don't satisfy the hypotheses of proposition 5.5. In particular, see the articles [Porn] of Pommersheim, [CS] of Cappell-Shaneson, [Mor] of Morelli and [KK] of Kantor and Khovanskii.

6. The Proof of (3.15)

Combining Theorem 4.2 with the results of §5 we obtain the following theorem:

Theorem 6.1. Let q be an element of the weight lattice in 9*, and assume that the following hypotheses are satisfied for all p E MG:

1. The linear isotropy representation of G on the tangent space at p is quasi-free.

r 6.1) 2. The weight vectors, q - </J(p) and q - </J(p) + '1';;' - 'Yp lie in the same connected component of the open cone, Ap-

Then, the multiplicity with which q occurs as a weight of the representa­tion ofG on g(M) is equal to the Riemann-Roch number of the pre-quantum line bundle, L q •

Concerning the hypotheses (6.1) we note: 1. For certain strategically situated weights these hypotheses can be weak­

ened. For instance if q lies close to an exterior vertex, </J(po), of the moment polytope, it suffices to assume (6.1) for Po (See the comments at the end of §2).

2. From the assumption that q - </J{p) E Ap for all p E M G , it is not hard to show that q is a regular value of the moment map. (However, the converse is not true.)

3. The assumption that the isotropy action of G is quasi-free at every fixed point is equivalent to the assumption that the action of G on M

Reduced phase spaces and Riemann-Roch 333

is quasi-free. (See [DGP].) Thus, in particular, if this hypothesis holds, Mq is a manifold for every regular value of the moment map. Finally we would like to come back to a comment which we made

in the introduction: our suggestion that one take as a definition of the quantization function

Q(M) = 7T!LM , (6.2)

and clarify what we mean by it. By the K-theoretic version of the Atiyah­Singer theorem one can attach to an elliptic operator, D, two elements of K(pt). The first, the analytic index of D, is just the virtual vector space

kernel(D) - cokernel(D) . (6.3)

The second, the topological index, is the K-theoretic push-forward

(d7T).e(u(D»

where d7T is the map, T* M - pt, and e(u(D» is an element of KcompacdT* M) defined by means of the symbol of D. (In particular it only depends on the homotopy class of the symbol. See [AS].) The punch­line of the index theorem, of course, is that these two definitions agree. Hence, in particular, if D = be, the analytic index, which is the element, (3.13), of K(pt) is equal to the topological index of be which is (morally, speaking) (6.2).14

References

[AB,l] M. Atiyah and R. Bott, The moment map and equivariant coho­mology. Topology, 23 (1984) 1-28.

[AB,2] , A Lefschetz fixed point formula for elliptic complexes II: Applications. Ann. of Math. 88 (1968) 451-91.

[AS] M. Atiyah and I.M. Singer, The index of elliptic operators: I. Ann of Math. 87 (1968) 484-530.

[BV] N. Berline and M. Vergne, Classes characteristiques equivariantes. Formule de localization en cohomologie equivariante. C.R.Acad. Sci. Paris 295 (1982), 539-541.

[CS] S. Cappell and J. Shaneson, Todd classes, toric varieties and counting lattice points. (to appear).

14 Even precisely speaking, if M is a complex projective variety. In this case Grothendieck's version of Hirzebruch-Riemann-Roch says that the analytic index of be is 7T!LM.

334 v. Guillemin

[Dan] V.I. Danilov, The geometry of to ric varieties. Russian Math. Surveys 33 (1978) 87-154.

[Del] T. Delzant, HamiItoniens periodiques et image convexe de l'application moment. Bull. Soc. Math. Prance 116 (1988) 315-339.

[DGP] R. DeSouza, V. Guillemin and E. Prato, Consequences of quasi-free. Ann. Global Anal. Geom. 8 (1990) 77-85.

[G] V. Guillemin, Kaehler structures on toric varieties (to appear in the J. of DiJJ. Geom).

[GLS] v. Guillemin, E. Lerman and S. Sternberg, On the Kostant multi­plicity formula. JGP5 3 (1988) 2-30.

[GS,I] V. Guillemin and S. Sternberg, Geometric quantization and multi­plicities of group representations. Invent. Math. 67 (1982) 515-538.

[GS,2] , Symplectic Techniques in Physics. Cambridge U. Press, Cambridge 1984.

[JK] L.C. Jeffrey and F.C. Kirwan, Localization for non-abelian group ac­tions. Reprint alg-geom/9307001 (1993).

[KK] M.M. Kantor and A. Khovanskii, Integral points in convex polyhedra, a combinatorial Riemann-Roch theorem and generalized MacLaurin formula. IHES (1992) M 932-37.

[KP] A. Khovanskii and S. Pukhlikov, Mesures additives invariantes pour les polyhedres virtuels. Alg. i analyz 4 (1992) 161-185.

[Mar] C.M. MarIe, Modele d'action hamiItonienne d'un groupe de Lie sur une variete symplectique. Rend. del Seminario Mat. Torino 43 (1985) 227-251.

[Mor] R. Morelli, A theory of polyhedra. Advances in Math. 97 (1993) 1-73.

[Porn] J. Pommersheim, Toric varieties, lattice points and Dedekind sums. Math. Ann. 93 (1993) 1-24.

Department of Mathematics, MIT, Cambridge, MA 02139

Received January 17, 1994

The Invariants of Degree up to 6 of all n-ary mr ics

Roger Howe l

1. Introduction

The problem which preoccupied invariant theorists of the 19th century was to describe the invariants of a form (Le., a homogeneous polynomial, or, dually and essentially equivalently, a symmetric tensor), of degree m in n variables - an n-ary m-ic. An invariant of degree d of an n-ary m-ic is an element of sd(sm(cn» , the d-th symmetric power of the m-th symmetric power of C n , which is invariant under the natural action of S L( n, C) on this space. The usual approach to the problem was to fix nand m, and to ask for a set of "fundamental invariants." This meant using the fact that the sum

Sm,n = L sd(sm(cn» d~O

(1.1)

has naturally the structure of algebra, e.g., the symmetric algebra on sm(cn ). The natural action of GL(n, C) on sm(cn ) extends to an ac­tion on Sm,n by algebra automorphisms, and the space S!~n of elements invariant under SL(n, C) is a subalgebra. By "fundamental invariants," one means a set of generators for this algebra. This problem has been solved explicitly for only a small number of pairs (n, m). According to [Di], the "known" cases are (n,2) for all nj (n,3) for n = 2,3, 4j and (2, m) for m ~ 8. See also [DL] , [Sh] for recent progress, and [Me] for the state of affairs in the late 19th century.

In [Ho 1], it is noted that the sum

L sd(sm(cn» = Ad,n (1.2) m~O

also carries an algebra structure, and the natural action of G L( n, C) on the summands of Ad,n gives an action by algebra automorphisms. Description of the subalgebra A~,~ of invariants in Ad,n would yield a description of the invariants of a given degree d for all forms in n variables.

There is a close relation between the two families of algebras, and the classical computations implicitly exploit the connection. The identification

1 Research partially supported by NSF Grant DMS-9224358

336 R. Howe

Sl(Sd(Cn» ~ Sd(sl(cn» ~ Sd(Cn ) of the degree one subspaces gives rise to a GL(n, C)-equivariant algebra homomorphism

(1.3)

by the universal property of the symmetric algebra [Ja]. When n = 2, the homomorphisms ad,2 are isomorphisms, by Hermite Reciprocity [Ho I], [KR]. Indeed, the statement that (1.3) is an isomorphism when n = 2 is the optimal expression of Hermite Reciprocity, which is usually formu­lated as a collection of GL(2, C)-isomorphisms between individual pairs of spaces: sd(sm(c2» ~ sm(sd(C2». As explained in (Ho I], the classical machinery of "bracket factors" in fact involves calculation inside Ad,2' One then exploits the isomorphism (1.3) to transfer information gained about Ad,2 back to Sd,2' The algebras Ad,n seem to be somewhat more directly accessible than the usually considered Sd,n' In particular, the analog of the rectangle (!R) on page 154 of [Ho 1] allows one to study A~~ via the action of a torus on a Grassmannian. We will follow this appro'ach to obtain a description of A~,~ for d $ 6 and all n.This will let us give a description of the invariants of degree 6 or less for an n-ary m-ic for arbitrary m and n.

Because of various degeneracies and symmetries, which are discussed in [Ho 2] and reviewed in Section 6, the only essentially new cases are (d,n) = (6,3) and (6,4). These are described in Section 4 and Section 5 respectively, while Sections 2 and 3 give preliminary calculations which may have some independent interest.

I am delighted to dedicate this article to Bert Kostant whose mathe­matical insight and taste have afforded me much pleasure.

2. Torus invariants for A3(C6 )

We will first determine the structure of A~,~. The approach to A~,~ by the analog of the rectangle (!R) of [Ho 1] involves describing the algebra of invariants for the diagonal torus of 8L(6, C) acting on the coordinate ring R(Gg) for the Grassmannian Gg of 3-planes in 6-space.

For the convenience of the reader, we write the relevant version of the rectangle (!R):

S(Cn ®Cd ) ;2 S(Cn ®Cd)A~ ;2 (S(Cn ® Cd)A~)Sd ~ Ad,n ul ul ul ul

S(Cn ® Cd)SL n ;2 (S(Cn ® Cd)SL n )A~ ;2 «s(cn ® Cd)SL n )A~ )Sd ~ ASL d,n II II II II

R(G~) ;2 R(G~)A~ ;2 ('R.(G~)A~)Sd ~ ASL d,n

The Invariants of Degreee up to 6 337

We want to find the ring in the lower right hand corner of this rectangle. Our approach is to proceed down the left hand side, then along the bottom row (rather than along the top row and down the right hand side).

Let A3(C6) be the third exterior power of C 6, and let PA3(C6) be its associated projective space. We can embed G~ in PA3(C6) in the usual way, as the subvariety of decomposable vectors, modulo scalars. Then n(G~) is just the restriction of polynomials on A3(C6) to the inverse image of G~ in A3(C6). This inverse image is a cone over G~.

We can in fact describe the invariants for the torus of 8L(6, C) in the full polynomial algebra p(A3(C6». We proceed to do this. It is essentially equivalent and slightly more convenient to work in the symmetric algebra S(A3(C6», so we will do this. Let A6 = A denote the group of 6 x 6 diagonal matrices, and let AA = A I denote the subgroup of matrices with determinant one. The standard basis for A3(C6) consists of the twenty elements

1::::; i < j < k::::; 6, (2.1)

the wedge products of triples of standard basis vectors for C 6 • These are eigenvectors for A6 • If a is a diagonal matrix with diagonal entries {aj : 1 ::::; j ::::; 6}, then

(2.2)

We want to compute the invariants for the action of A I on the symmetric tensors on A3(C6). Since the [i,j,k] are eigenvectors for AI, the space of all invariants for Al will be spanned by those monomials in the [i,j, k] which are invariants for AI. Under multiplication, the set of invariant monomials will form a semigroup. We will describe a set of generators for this semigroup.

A monomial II Ii, j, k]m;jk

{i,j,k}

will be an A I-invariant precisely when the sums

Se = L mijk

eE{i,j,k}

(2.3)

(2.4)

are all equal. In particular, when the Sf all equal one, we have the monomial

[1,2,3][4,5,6], (2.5)

and also the transforms of it by any permutation of the standard basis elements ej. We can identify the group of these permutations with the symmetric group 86. There are ten transforms of element (2.5) by 86.

When Sf = 2, we have products of the elements of type (2.5), and also

[1,2,3][1,4,5][2,4,6][3,5,6] (2.5)

338 R. Howe

and all its transforms by 86 , There are 30 of these. When St = 3, we have 3-fold products of the elements of type (2.5) and products of an element of type (2:5) with one of type (2.6), and in addition we have elements obtained from

[1,2,3][1,2,4][3,4,5][3,4,6][1,5,6][2,5,6] (2.7)

by the action of permutations. There are again 30 of these.

Theorem 2.8. The algebm S(A3(C6»Al is genemted by the 86 tmnsforms of the elements (2.5), (2.6) and (2.7).

Proof. We need to show that any monomial of the form (2.3) in which the exponents mijk satisfy the relations (2.4) contains a factor of one of the types (2.5), (2.6) or (2.7). Since each element [i,j, k] involves three indices and there are six indices all together, we see that the equations (2.4) imply that

1 St = 2 L mijk·

{i,j,k}

In other words, a given label i, 1 :5 i :5 6, will occur in exactly half of the factors of (2.3). It follows that, for a pair {i,j} of labels, the number of triples (counted with multiplicity given by their exponent) in which i and j appear together equals the number of triples from which both i and j are absent. This observation is basic to our argument.

Consider a monomial of the form (2.3). Up to permutation by an element of 86 , we may assume that m123 > 0, that is, [1,2,3] appears as a factor of (2.3). If [4,5,6] also appears as a factor, then we have a factor of type (2.5). So assume that [4,5,6] does not appear. Since none of 4,5, or 6 is involved in [1,2,3], each of the pairs {4,5}, {4,6} and {5,6} must be involved in some factor of (2.3). That is, we have factors of the form [x,4,5], [y,4,6] and [z,5,6], where {x,y,z} ~ {1,2,3}. If we can choose x, y, and z to be 1,2, and 3 in some order, then we have a factor of type (2.6), so we may assume this is impossible. We consider various cases.

Case i) x = y = z; that is, there is only one choice for each of x, y and z, and these choices are all the same. Up to action by 86, we may assume that x = y = z = 1. Thus we have the factors [1,2,3], [1,4,5], [1,4,6] and [1,5,6]. Other factors must involve only one element from {4, 5, 6} and two elements from {I, 2, 3}. There must be at least one factor which does not involve 1, or clearly S1 will be larger than S2 or S3' Thus we must have a factor [2,3, x], where x is in {4, 5, 6}. Comparing this with the factors we already have, we see that in this case, the product (2.3) has a factor of type (2.5).

Case ii) Assume that {x,y,z} only comprises two distinct elements. Up to the action of 86 , we may assume that these are 1 and 2. More specifically, we may assume that we have as factors of (2.3) the ele­ments [1,2,3], [1,4,5], [1,4,6] and [2,5,6], and no factors [3, u, v] with

The Invariants of Degreee up to 6 339

{u,v} ~ {4,5,6}. On the other hand, for each u in {4,5,6}, neither ele­ment of the pair {3, u} is involved in some one of the factors [1,4,5], [1,4,6] or [2,5,6]. Hence we must have factors [x, 3, 4], [y, 3, 5] and [z, 3, 6] with x, y and z each equal to 1 or 2. To avoid factors of type (2.5), we see that we must choose x = 2, y = 1, and z = 1. From among the factors we have so far, we can form the product [2,3,4][2,5,6][1,3,5][1,4,6], which is of type (2.6).

Case iii) We assume that all of {I, 2, 3} may appear as x, y or z, but that we cannot match three distinct x, y and z with the three pairs {4, 5}, {4, 6} and {5,6}. Up to action by 86 , this means we an assume we have factors [1,4,5], [1,4,6], [2, 5, 6] and [3,5,6]. The factor [1,5,6] may also occur, but the factors [x, 4, 5] or [x, 4, 6] with x equal to 2 or 3, should not occur. Since the factor [2,5,6] involves neither of 3 or 4, we must have a factor [x, 3, 4]. Since x should not equal 5 or 6, it must be 1 or 2. In order to avoid the factor [2,5,6][1,3,4]' which is of type (2.5), we are forced to take x = 2, and so we have the factor [2,3,4]. From the factors we have found so far, we can form the product [1,2,3][2,3,4][1,4,5][1,4,6][2,5,6][3,5,6]' which is of type (2.7).

Thus in all cases we have found a factor of one of the required types, so Theorem 2.8 is proved.

3. Torus invariants in R(Gg)

Our main object of study is not the full algebra of Al invariant polynomials on A3(C6), but rather the algebra R(Gg)Al of Al invariants in the coordi­nate ring R(Gg). We can combine Theorem 2.8 with further information to compute R(Gg)Al rather precisely.

Recall the structure of R(Gg) as module for GL6 . We use diagram notation to describe the representations of GLn; see for example [Ho 1,3]. We will have to deal primarily with representations corresponding to dia­grams with three rows. We denote a representation of GLn corresponding to a diagram with row lengths (a, b, c) (where a ~ b ~ c are non-negative integers) by p~a,b,c). It is well known (see for example [Ho 3]) that as G L6 module, the ring R(Gg) decomposes as the sum

00 00

R(G~) = LR(G~)a ~ LP~a,a,a). (3.1) a=O a=O

Thus R(Gg)Al decomposes as the sum of the "zero-weight spaces" (p~a,a,a»Al. A zero weight space (p~a,b,c»Al can only be non-zero if a+b+c

is divisible by 6. Thus (p~a,a,a»Al = 0 unless a = 2b is even.

Proposition 3.2. The dimension (p~2b,2b,2b»Al is equal to

(b~4) + (b~2).

340 R. Howe

Here (~) is the usual binomial coefficient which counts k-element subsets

of a set of size n.

Proof. The Branching Law from GLn to GLn _ 1 (see for example [Ho 3]) says that the restriction of p~2b,2b,2b) to GL5 consists of representations with diagram (2b, 2b, c) for 0 :5 c :5 2b. Of these, the only one which can contribute to the zero weight space corresponds to the diagram (2b, 2b, b). Thus the dimension of the zero weight space in p~2b,2b,2b) is the same as in

p~2b,2b,b). If we further restrict to GL4 , the representations we get which can contribute to the zero weight space are those corresponding to the diagrams (2b, 2b - j,j), for 0 :5 j :5 b. We have one copy of each of these. From [Fo] or [Ho 2], we have the formula

Summing these numbers for 0 :5 j :5 b gives the stated value for ( (2b,2b,2b»Al P6 .

In particular, the dimension of (p~2,2,2»Al is 5. It was shown by Kostant [Ko] (see also [Gu], [AMT]), that for a diagram D of total weight n, the zero weight space (p!?)A 1 in the representation of GLn corresponding to D is the irreducible module for 8n corresponding to the same diagram. (This fact plays a rather interesting role in invariant theory [Ho 3].) Thus, the space (p~2,2,2»A 1 defines the 5-dimensional irreducible module for 8 6

corresponding to the diagram (2,2,2). We observe that (b ~ 4) is just

the dimension of the space of polynomials of degree b in 5 variables. This suggests the possibility that the structure of n(Gg)Al is as described in the following theorem.

Theorem 3.3. The five dimensional subspace (p~2,2,2»Al generates a five

variable polynomial subalgebra ofn(Gg)Al. FUrther, we have a decompo­sition

(3.4)

where the element f3 transforms by the sign character of 86 • Moreover, f3 satisfies a quadratic equation over S(p~2,2,2»Al , and we have the direct sum decomposition

Proof. It is well-known (see for example [HP]) that the kernel of the restriction map from p(A3(C6» to n(Gg) is generated as an ideal by the

The Invariants of Degreee up to 6 341

quadratic polynomials known as the Pliicker Relations. These consist of all G L6 transforms of the element

[1,2,3][1,4,5] + [1,2,4][1,5,3] + [1,2,5][1,3,4]. (3.6)

In particular, they include all 86 transforms of this element. From Theorem 2.8, we know that the algebra P(A3(C6»A1 is gener­

ated by its elements of degrees 2,4 and 6, which are described explicitly by formulas (2.5), (2.6) and (2.7). It follows that the algebra 'R.(G~)AI is gen­erated by the images of these elements under the restriction map. (Actually, the elements (2.5), (2.6) and (2.7) represent elements of the symmetric al­gebra, and likewise, the Pliicker element (3.6) is written as an element of the symmetric algebra. For explicit computations, it is convenient to work in the symmetric algebra, while in speaking of the homomorphism to the coordinate ring, it is convenient to talk in terms of the polynomial ring. We hope this somewhat cavalier failure to distinguish carefully between the two mutually dual algebras p(A3(C6» and S(A3(C6» will not confuse the reader.) Thus to determine the structure of 'R.(G~)AI , we should consider the additional relations imposed among the elements (2.5), (2.6) and (2.7) by the Pliicker Relations.

Consider the element (2.6). If we use the element (3.6) to rewrite the first two factors of (2.6), we obtain -[1,2,4][1,5,3][2,4,6][3,5,6] -[1,2,5][1,3,4][2,4,6][3,5,6]. The first term of this expression is a product of two factors of the type of (2.5). The second is of type (2.6). Precisely, it is the transform of (2.6) by the permutation 835 which exchanges 3 and 5. This says that (2.6) and its transform by 835 are equal (up to sign) in 'R.(G~) modulo the subalgebra generated by elements of type (2.5). Ap­propriate permutations of the element (3.6) lead to a similar conclusion for the transpositions 836,831. 821. 841. 842, 846, 826 and 856. It is easy to check that these transpositions generate the whole group 86. Hence mod-

(222) Al . ulo the products of elements of (P6 " ) ,there IS only one more element in (p~4,4,4»AI. We will call this element (3. It must be an eigenvector for 86. Careful attention to signs in the computation above shows the eigen­character of 86 defined by (3 is the sign character. We note also that since dim(p~4,4,4»AI = 16 (by Proposition 3.2), and dimS2((p~2,2,2»Al) = 15, there can be no quadratic relations among the elements of (p~2,2,2»Al , and the direct sum decomposition (3.4) must hold.

Further, since the image in 'R.(G~) of element (2.6) is equivalent modulo S2((p~2,2,2»Al) to its 86 transform [4,5,6][2,3,6][1,3,5][1,2,4], and the product of this with (2.6) is a four-fold product of elements of type (2.5), the element (3 must satisfy a quadratic equation over the algebra generated b ( 2,2,2»AI

y P6 . Next consider elements of type (2.7). If we use the appropriate cousin

of relation (3.6) to rewrite the factor [1,2,3][1,5,6]' we see that in 'R.(G~),

342 R. Howe

the element (2.7) is equal to

[1,2,5][1,2,4][3,4,5][3,4,6][1,3,6][2,5,6]

- [1,2,6][1,2,4][3,4,5][3,4,6][1,3,5][2,5,6].

This is a sum of products of elements of type (2.5) and (2.6). Thus, in R(Gg), the elements of type (2.7) are already in the algebra generated by the elements of degrees 2 and 4, more precisely by (p~2.2.2»)Al and by {3.

From what we have shown so far, we can conclude that R(Gg)Al is some quotient of the algebra on the right hand side of equation (3.5). If it were a proper quotient, its homogeneous components could not have the dimensions specified by Proposition 3.2, since these are exactly the dimensions of the components of the right hand side of (3.5). Thus equation (3.5) must hold, and the theorem is proved.

Remark. I thank a referee for pointing out that Theorem 3.3 can also be found in [DO], pp. 17-20. The calculation of [DO] is broadly similar, though it differs in detail. In place of Theorem 2.8 and (3.6), [DO] use standard monomial theory [HP]. In place of Proposition 3.6, they state the result of a lengthy ad hoc calculation. They describe an explicit equation for {3. In view of Section 4 below, this equation is almost, but not quite, determined a priori (since 84((p~2.2.2»)Al )S6 has dimension 2). They also

compute the algebra R(G~)Al. It should also be remarked that [DO] cites [Co] as a source for most of its results.

4. The algebra A~.~

According to the scheme of the rectangle (Vt), described at the start of Section 2, we can obtain the algebra A~~ by computing the 86 invariants in the algebra R(Gg)Al: .

(4.1)

The invariants of the standard representation of 8n are described by La­grange's theorem. They are generated by the elementary symmetric func­tions Uk for 2 :$ k :$ n. The invariants of the other representations of 8n are hard to describe and are typically unknown. However, the representation r(2, 2, 2) of 86 parametrized by the diagram (2,2,2) is a happy exception to this rule. It is known that, alone among symmetric groups, 86 allows an outer automorphism ([Hup], pp. 175-6). It can be checked that this outer automorphism, exchanges the standard permutation representation with r(2, 2, 2). Thus the invariants of r(2, 2, 2) must be a polynomial algebra on generators of degrees 2, 3, 4, 5 and 6, just as for the permutation represen­tation. Since the element {3 of equation (3.5) is an eigenvector for 86 , with

The Invariants of Degreee up to 6 343

the sign character as eigencharacter, the 86 invariants in (3S(p~2,2,2»Al will be of the form (3q, where q is an element of S((p~2,2,2»Al) which also trans­forms by the sign character of 86 , Since in P(C5 ), where C 5 indicates the standard permutation representation of 86, the sign eigenvectors are generated as a module by a single element, the discriminant, of degree 15, we conclude that

((3S((p~2,2,2»Al))86 = (3.&(S((p~2,2,2»Al)86),

where .& is the analog in S((p~2,2,2»Al) of the discriminant in P(C5). Par­allel to the situation for p(C5), the element .& will have its square in S( (p~2,2,2»A 1 )86 • Hence so also will the element (3.&. Thus we have a direct sum decomposition

We state formally some consequences of this description.

Proposition 4.3. The algebra Ai~ is generated by six elements, of degrees 2,3,4,5,6 and 17. The first five el~ments are algebraically independent, and the last one satisfies a quadratic equation over the algebra generated by the first five. Hence the Hilbert series of Ag is equal to ,

(4.3)

This is of course the generating function for the dimensions of the spaces s6(sm(C3))8L of degree six invariants for the ternary m-ics. Our description of the algebra structure will give explicit basis elements for these spaces of invariants.

5. The algebras Ai,~ and Ai,~

We will determine the algebras Ai~ and Ai~ by an approach similar to that of Section 4. By Hermite Reciprocity, the algebra Ai~ describes the invariants of a binary sextic, which were determined in the 19th century [Me]. Our computation will use the set-up of [Ho 1], and will have the advantage that it will provide also a description of Ai,~.

The main point is to describe n(G~)Al as an 86 module. For this, first write

00 00

n(G~) ~ En(G~)a ~ Ep~a,a), (5.1) a=O a=O

in analogy with equation (3.1). By Kempe's Theorem [Ke], [Ho 1], [KR] we know that n(G~)Al is generated by (n(G~)3)Al ~ (p~3,3»Al. Hence

344 R. Howe

R(G~)Al is some quotient of the symmetric algebra S«p~3,3»A\ Accord­ing to the result of Gutkin-Kostant [Gu], [Ko] , [AMT], [Ho 3], the action of 86 on (R(G~)3)Al defines the irreducible module T(3, 3) corresponding to the diagram (3,3). This is the twist by the sign character of the module T(2, 2, 2) studied in Section 4. We know from Section 4 that S( T(2, 2, 2» has an algebra of invariants S( T(2, 2, 2) )86 generated by elements of order 2, 3, 4, 5 and 6, and that its sign-eigenspace is generated as a module for S( T(2, 2,2) )86 by an element Li of degree 15. Let us denote the generators of S(T(2, 2, 2»86 by "Ii for 2 :$ j :$ 6. Then we can write

S( T(2, 2, 2»86 >:::J C("/2, "13, "14, "Is, "16), (5.2)

and S( T(2, 2, 2»86 ,sgn >:::J Li C("/2, "13, "14, "Is, "16).

If we twist T(2, 2, 2) by the sign character, the effect on S(T(2, 2, 2» will be to turn invariants of odd degree to sign-eigenvectors and vice versa, while invariants of even degree will remain invariant. To distinguish between even and odd degree in the formulas (5.2), we write

(5.3)

and S( T(2, 2, 2) )86 >:::J B ffi "I3B ffi "IsB ffi "I3"1SB

and S( T(2, 2, 2) )86 ,sgn >:::J LiB ffi Li"l3B ffi Li"lsB ffi Li"l3"1sB.

If we use the isomorphism T(3, 3) >:::J T(2, 2, 2) ® sgn to identify the space of T(3,3) with the space of T(2, 2, 2), we find from (5.3) that

S(T(3,3»86 >:::J B ffi "I3"1SB ffi LiB ffi Li"l3"1sB (5.4)

and S( T(3, 3) )86 ,sgn >:::J "I3B ffi "IsB ffi Li"l3B ffi Li"lsB.

The algebra (R( G~)A 1 )86 will be some quotient of the first line of for­mula (5.4). If we use the dimension formulas from [Ho 1], §5.9, we see that dim(R(G~)3)Al = 5 = dim(T(3,3», and dim(R(G~)6)Al = 15 = dimS2(T(3,3», but dim(R(G~)9)Al = 34 = dim S3(T(3, 3» - 1. Thus, in S3(T(3,3» there is an element which projects to zero in R(G~)Al. From formulas (5.4), we see that the unique 86-eigenvector in S3( T(3, 3» is "13. Let 1("/3) be the ideal generated by "13. The quotient algebra S(T(3,3»/1("/3) inherits a grading from S(T(3,3». We denote the a-th homogeneous component by (S(T(3,3»/1("/3»a. We compute

dim(S(T(3, 3)/1 ("/3))) a = (a 14) _ ( a 11 )

= 3 (a; 2) + a + 1 = dim(R(G~t)Al.

The Invariants of Degreee up to 6 345

The last of these equalities comes from [Ho 1], §5.9. See also [Ta]. It follows from this equality of dimensions that

(5.5)

Denote by B the quotient of B by "Y~. Thus

(5.6)

It follows from (5.4) and (5.5) that

('R.(G~)Al)S6 ~ B E& LiB. (5.7)

As we have noted, the algebra ('R.(G~)Al)S6 is essentially the algebra of invariants of the binary sextic. However, there is an issue of grading which should be discussed. The algebra Ad,n is of course graded as in equation (1.2), and A~.~ inherits this grading. The algebra «'R.(G~»A~ )Sd, however, has two possible gradings. There is the grading implicit in the analog of decomposition (3.1)

00

'R.(G~) ~ L'R.(G~)a, (5.8) a=O

where 'R.(G~)a is an irreducible GLd module equivalent to the a-th Cart an power of An(cd). This corresponds to a diagram with d rows of length a. On the other hand, a character of the torus Ad which is trivial on Aj will be a power of the (restriction to Ad of the) determinant character. We could also grade ('R.(G~»A~ according to the powers of det under the action of Ad' The relation between the two gradings is: elements of ('R.(G~)a)A~ transform under Ad according to detm where md = an. Thus the space sd(sm(cn»SL is identified to «'R.(G~)a)A~)Sd, with a = md/n.

When (d,n) = (6,2), this says a = 3m. Since r(3,3) ~ ('R.(G~)3)Al, the grading implicit in equation (5.7) is the grading of equation (5.8) di­vided by 3. Hence it coincides with the natural grading on Ag.~. We thus have the following result.

Proposition 5.9. The algebra Ag~ has a set of generators of degrees 2, 4,6,10 and 15. The first four ge";'erators are algebraically independent, generating a polynomial algebra fJ of Ag.~. The degree 15 generator has its square in fJ, giving a decomposition of Ag.~ as on the right hand side of formula (5.7).

As a check, we note that the Hilbert function of the right hand side of (5.7) is

1 + t l5

346 R. Howe

which is indeed the Hilbert series for the invariants of the binary sextic (see for example [Ho 1], §5.9.4).

To compute A~,~, we use the duality between A~,~ and A~,~_n (cf. [Ho 2], §4.3). This derives from the GLd module isomorphisms

(5.10)

Here X* indicates the contragredient to a module X. From (5.1O) we deduce an isomorphism of Sd modules

(5.11)

Here we may drop the * because all Sd-modules are self dual. Taking into account that the sign character sgn is of order two, we may write the isomorphisms (5.11) slightly more concretely as follows

(5.12)

and

If we twist 7{3, 3) with sgn, we return to 7{2, 2, 2). Combining equations (5.5) and (5.10), we conclude that

(5.13)

Combining this with equation (5.4), we get

(5.14)

This is actually the polynomial algebra in generators "Ij for j = 2,4,5 and 6.

Taking into account the relation between the various gradings involved, we see that we should double the grading implicit in equation (5.14) in order to arrive at the natural grading an A~,~. Hence we obtain the following description.

Proposition 5.15. The algebra A~,~ is a polynomial algebra on generators of degrees 4,8, 10 and 12. It has Hilbert series

1

The Invariants of Degreee up to 6 347

6. Concluding remarks

In Sections 4 and 5, we have described the algebras A~,~ for n = 2, 3 and 4. The algebras A~,~ for n = 1,5 and 6 are very easy to describe, and may be read off from the remarks in [Ho 2], §4.3. For n > 6, the algebras A~,~ are trivial. Thus the computations given here essentially determine A~~ for all n, which is to say, they essentially determine the degree 6 invariants of the n-ary m-ic for all nand m. The algebras A~,~ for d ::; 4 are also very easy to determine. They all can be found from the remarks of [Ho 2], §4.3, except for Af~, which is the algebra of invariants of the binary quartic. It has a simple ~tructure which was found in the 19th century; it is described in [Ho 1] §5.9.3, among other places.

For d = 5, the only two non-trivial cases are n = 2 and n = 3. The algebra An is isomorphic by Hermite Reciprocity to the invariants of the binary 'quintic. It has generators of degrees 4,8, and 12, which are algebraically independent, and a generator of degree 18 which is quadratic over the first 3 [Di] , [Sp]. It is realized as a quotient of the 85 invariants in S«p~5,5»A\ The space (p~5,5»Al is six-dimensional, and is irreducible as an 85 module. It is a realization of the 85 module corresponding to the diagram (3,1,1). Since this representation is isomorphic to its twist by sign, we conclude that the algebra Ag~ is isomorphic to Ag~; however, because of the different gradings involved, its generators will have degrees 6, 12, 18 and 27 rather than 4, 8, 12 and 18. It is of interest to give a description of Ag,~ and Ag,~ parallel to the calculations of Sections 4 and 5; we hope to return to this topic.

Thus by combining the calculations given here with easy facts and classical results for binary forms, we can describe sd(sm(cn » for d ::; 6 and all m and n. For d > 6, one will run into the same types of difficulties that ended the golden age of computation in the 19th century.

References

[AMT] S. Ariki, J. Matsuzawa, and 1. Terada, Representations of Weyl groups on zero weight spaces, in: Algebraic and Topological Theories, to the memory of Dr. Takehiki Miyata, Kinokuniya Co. LTD, Tokyo 1985, pp. 546-568.

[Co] A. Coble, Algebraic Geometry and Theta Functions, CoHo Pub. to, (3rd ed. 1969), American Mathematical Society, Providence, R.1. 1929.

[Di] J. Dixmier, Quelques aspects de la thoorie des invariants, Gaz. des. Maths. 43 1990, 38-64.

[DL] J. Dixmier and D. Lazard, Le nombre minimum d'invariants fon­damentaux pour les formes de degre 7, Port. Math. 43 1985-86, 377-392.

348 R. Howe

[DO] I. Dolgachev and D. Ortland, Point sets in projective spaces and theta functions, Asterisque 165 1988.

[Fo] H. Foulkes, Plethysms of S-functions, Trans. Roy. Soc. Lon. A246 1954, 555-593.

[Gu] E. Gutkin, Representations of the Weyl group in the space of vectors of zero weight, (in Russian): Uspehi. Mat. Nauk 28 1973, 237-238.

[HP] W. Hodge and D. Pedoe, Methods of Algebmic Geometry I, Cam­bridge University Press, Cambridge, UK 1947.

[Ho 1] R. Howe, The Classical Groups and Invariants of Binary Forms, in: The Mathematical Heritage of Hermann Weyl, (R. Wells, ed.), Proc. Symp Pure Math. 48 1988, American Mathematical Society, Providence, R.I., pp. 133-166.

[Ho 2] R. Howe, (GLn' GLm)-duality and symmetric plethysm, Proc. In­dian Acad. Sci. (Math. Sci.) 97 1987, 85-109.

[Ho 3] R. Howe, Perspectives on Invariant Theory: Schur Duality, Multiplicity-Free Actions and Beyond, to appear Ismel Mathemati­cal Conference Proceedings.

[Hup] B. Huppert, Endiche Gruppen I, Grund. Math. Wiss. 134, Springer-Verlag, Berlin, Heidelberg, New York 1967.

[Ja] N. Jacobson, Basic Algebm II, W. H. Freeman, New York 1980. [Ke] A. Kempe, On regular difference terms, Proc. Lond. Math. Soc. 25

1894, 343-359. [KoJ B. Kostant, On Macdonald's 17-function formula, the Laplacian and

generalized exponents, Adv. in Math. 20 (1976), 257-285. [KR] J. Kung and G.-C. Rota, The invariant theory of binary forms,

B.A.M.S. (N.S.) 10 1984, 27-85. [Me] W. Meyer, Bericht iiber den gegenwiirtigen Stand der Invarianten­

theorie, D.M. V. 1 1892, 79-292. ISh] T. Shioda, On the graded ring of binary octavics, Am. J. Math. 89

1967, 1022-1046. [Sp] T. Springer, Invariant Theory, Lecture Notes in Math. 585,

Springer-Verlag, Berlin, New York 1977. [Ta] T. Tambour, An explicit formula counting non-commutative clas­

sical invariants, C.R. Math. Rep. Acad. Sci. Canada IX 1987, 183-188.

[Th] R. Thrall, On symmetrized Kronecker powers and the structure of the free Lie ring, Am. J. Math. 64 1942, 371-388.

Department of Mathematics, Box 2155, Yale Station, Yale University, New Haven, CT 06520-2155, USA.

Received April 21, 1994

Tensor Products of Modules for a Vertex Operator Algebra and Vertex Tensor Categories

Yi-Zhi Huang and James Lepowsky

Dedicated to Bert Kostant on the occasion of his sixty-fifth birthday

1. Introduction

In this paper, we present a theory of tensor products of classes of modules for a vertex operator algebra. We focus on motivating and explaining new structures and results in this theory, rather than on proofs, which are be­ing presented in a series of papers beginning with [HL4] and [HL5]. An announcement has also appeared [HLI]. The theory is based on both the formal-calculus approach to vertex operator algebra theory developed in [FLM2] and [FHL] and the precise geometric interpretation of the notion of vertex operator algebra established in [HI].

Recently, mathematicians have been more and more attracted to con­formal field theory, a physical theory which plays an important role in both condensed matter physics and string theory. Much of the research on conformal field theory has been centered on the conformal field theo­ries determined by holomorphic fields of weight I-the theories associated with certain highest-weight representations of affine Kac-Moody algebras. Many important structures and concepts which have arisen in recent years are related to this special class of conformal field theories-Wess-Zumino­Novikov-Witten (WZNW) models, the Knizhnik-Zamolodchikov equations and associated monodromy, quantum groups, braided tensor categories, the Jones polynomial and generalizations, three-manifold invariants, Chern­Simons theory, the Verlinde formula, etc. (see for instance [Will, [KZ], [J], [KI], [K2] , [Ve] , [TK] , [MS] , [Wi2] , [TUY] , [Drl] , [Dr2], [RT], [SV] , [Va], [KLI]-[KL5], [Fi], [Fa]). But there are also other important mathemati­cal structures that can be studied as conformal-field-theoretic structures, in particular, highest weight representations of the Virasoro algebra, W­algebras and their representations, and most particularly for us, the moon­shine module Vb for the Fischer-Griess Monster sporadic finite simple group [Gr] constructed in [FLMI] and [FLM2]. For the conformal field theories associated with these structures, there are no nonzero holomorphic fields of weight 1, and correspondingly, the special methods for studying those con­formal field theories associated with affine Lie algebras do not apply. We must have a broader viewpoint than that of this special class of conformal

350 Y.-Z. Huang and J. Lepowsky

field theories. The theory of vertex operator algebras, developed in [Bt], [FLM2] and

[FHL] (the research monograph [FLM2] also includes a detailed exposition of the theory, with examples), provides such a framework. Vertex operator algebras are "complex analogues" of both Lie algebras and commutative associative algebras. All the structures mentioned in the preceding para­graph can be and have been illuminated by the representation theory of vertex operator algebras. One important class of vertex operator algebras, generated by vectors of weight t, is constructed from certain highest weight modules for affine Lie algebras, and certain other highest weight modules can be given the structure of modules for these vertex operator algebras (see for example [FZ], [DL]). The theories of highest weight representa­tions of the Virasoro algebra and of representations of W-algebras can also be studied in terms of the representation theory of the corresponding vertex operator algebras.

The moonshine module vq [FLMI] is in fact an early example of ver­tex operator algebra, except that a full vertex operator algebra structure on Vb was stated to exist only later, by Borcherds in the announcement [Bt], in which the notion of "vertex algebra" was introduced. In these terms, the structure detailed in the announcement [FLMt] was the Monster­invariant generating weight-two substructure, equipped correspondingly with a "cross-bracket" operation (rather than the Lie bracket operation), of the vertex operator algebra structure on V q• The proof of the properties of the construction of Vb and of the action of the Monster on it [FLMt] was given in [FLM2], along with the construction of a Monster-invariant vertex operator algebra structure on it (stated to exist in [Bt]) , and an axiomatic study of the concept of vertex operator algebra was presented in the monograph [FHL]. Meanwhile, Belavin, Polyakov and Zamolodchikov [BPZ] and other physicists also introduced the basic features (on a phys­ical level of rigor) of what mathematicians came to understand as vertex operator algebras.

The most natural viewpoint for us is the viewpoint based on what we call the "Jacobi identity" (see [FLM2], [FHL] and the exposition [Lt]) for vertex operator algebras. A "complex analogue" of the Jacobi identity for Lie algebras, this identity is the main tool in the formal-calculus approach to the theory of vertex operators algebras developed in [FLM2] and [FHL] and used in the present work. It can be deduced from the physically-formulated axioms in [BPZ] or from the axioms for vertex algebras in [BI]. The no­tion of vertex operator algebra of [FLM2] and [FHL] is in fact a variant of the notion of vertex algebra of [Bt], equipped with the Jacobi identity as the main axiom and with certain grading restrictions assumed. It is this variant that we need. The Jacobi identity expresses an infinite family of generalized commutators (products like the Z-algebra products of [LW] or

Vertex tensor categories 351

the cross-brackets of [FLMl] mentioned above) in closed form via a gen­erating function based on formal delta-functions. In particular, the Jacobi identity exhibits the Lie-algebra-like properties of a vertex operator algebra rather than its associative-algebra-like properties, which are implicit in the operator-pro duct-expansion formalism in the physics literature, including [BPZ], and in the axioms for vertex algebras in [Bl].

The structure Vb is a vertex operator algebra which is simple (irre-ducible as a module for itself) such that

1. the rank (central charge) of Vb is 24,

2. there are no nonzero vectors of weight 1 in Vb,

3. Vb is its only irreducible module.

All these properties were established in [FLM2], except for the third, which was conjectured in [FLM2] and proved in [Do]. It was also conjectured in [FLM2] (and it remains a conjecture) that that these three properties characterize the vertex operator algebra Vb. We know that on the one hand,

1. ([FLMl], [FLM2]) the automorphism group of Vb is the Monster,

while on the other hand,

2. [Do] every Vb-module is a direct sum of copies of Vb itself (cf. (3) above), and in particular, the fusion algebra of Vb is the trivial one­dimensional associative algebra and all the fusing and braiding matri­ces are trivial for the associated holomorphic conformal field theory.

We have a curiously unsatisfactory, or rather, unstable, situation-the same structure Vb is the simplest possible from one viewpoint (that of monodromy of correlation functions) and the richest possible from another (that of symmetry); the Monster is arguably the most exceptional symme­try group that nature allows (cf. [Go]). We need a general theory to unify these two phenomena. (This problem was emphasized in [L2].) The theory of tensor products of modules for a vertex operator algebra, discussed in this paper, is being developed with this in mind.

Along with the algebraic theory of vertex operator algebras, our theory also has a geometric foundation: the techniques and results entering into the geometric interpretation of the notion of vertex operator algebra, in terms of a certain moduli space of multipunctured Riemann spheres with local coordinates vanishing at the punctures and the appropriate sewing operation on this moduli space, carried out in [HI]. The idea is to exploit fully the conformal structure implicit in the theory of vertex operator al­gebras, which can be thought of as the main ingredient of conformal field

352 Y.-Z. Huang and J. Lepowsky

theory. A more conceptual reformulation of this geometric interpretation has been accomplished (see [HL2], [HL3]) with the help of the notion of operad, which was originally introduced ([SI), [S2) , [M]) in a topological context in connection with the homotopy-theoretic characterization of loop spaces.

In the representation theory of Lie algebras, we have the classical notion of tensor product of modules, providing the conceptual foundation of the Clebsch-Gordan coefficients. The tensor product operation in the category of modules for a Lie algebra gives a classical example of a symmetric tensor category. Module categories for not-necessarily-cocommutative quantum groups (Hopf algebras) are sources of more general braided monoidal cat­egories, which give rise to braid group representations and knot and link invariants; see in particular P), [Kl), [K2) , [Drl], [Dr2) , [RT). For vertex operator algebras, we also have the notions of modules, intertwining oper­ators ("chiral vertex operators") among triples of modules and fusion rules analogous to those for Lie algebras (see [BPZ], [FS), [KZ), [TK) , [Vel and [MS]). The precise notion of intertwining operator that we need for the general theory, and that suggests the clearest analogy between the repre­sentation theory of vertex operator algebras and the representation theory of Lie algebras, is the one defined in [FHL), based on the Jacobi identity axiom for vertex operator algebras, rather than the more-commonly-used notion of chiral vertex operator based on certain Lie algebra coinvariants. One can ask whether there is a conceptual notion of tensor product of mod­ules for a vertex operator algebra, providing a foundation for the notion of fusion rules, by analogy with the case of Lie algebras. The Jacobi identity axiom suggests a kind of "complex analogue" of a Hopf algebra diagonal map, but it turns out that a considerable amount of work is needed to make this idea precise.

Motivated partly by this point of view and partly by the announce­ment [KLl) of Kazhdan and Lusztig, conceptually constructing a tensor product operation and establishing its basic categorical properties for cer­tain categories of modules for affine Lie algebras, we have introduced a general, conceptual notion of tensor products of modules for a suitable vertex operator algebra ([HLl), [HL4], [HL5]) , and in a continuing series of papers, we are establishing its basic categorical properties. In place of the braided monoidal categories that arise from the Kazhdan-Lusztig con­struction ([KLl)-[KL5]) and its extension to the WZNW-model case by Finkelberg [Fi), the result is instead "vertex tensor categories," which are categories equipped with a suitably commutative tensor product operation controlled by the sewing of elements of the moduli space mentioned above rather than by the traditional commutative diagrams of coherence theory for monoidal categories. What we have is in some sense a conceptual "com­plexification" of the notion of symmetric monoidal category, in that the

Vertex tensor categories 353

circle operad (cf. [HL2], [HL3]), which may be thought of as giving rise to the notion of symmetric monoidal category, is being replaced by a (partial) operad based analogously on the Riemann sphere. Moreover, a systematic specialization process yields an ordinary braided monoidal category from the vertex tensor category, for a suitable vertex operator algebra. This category is the usual braided monoidal category associated with the mon­odromy of correlation functions (essentially products of intertwining oper­ators) and the braiding and fusing matrices, giving the connection with the representation theory of quantum groups, as in [KLl]-[KL5], [Fi], and with knot and link invariants. That is, the familiar and fundamental topologi­cal information generated by holomorphic conformal field theory at genus zero now becomes a specialization of a theory starting from an underlying conformal geometry.

Our approach is based on the general concepts of vertex operator al­gebra theory rather than the methods of [KLl]-[KL5] and [Fi], which use special properties of affine Lie algebras, and as we have mentioned, we need to use the notion of intertwining operator based on the Jacobi identity rather than the notion based on Lie algebra coinvariantsj the two notions are indeed equivalent for the WZNW model and related models. Using this identity, we introduce a canonical notion of tensor product of modules for a suitable vertex operator algebra, defined in terms of an appropriate univer­sal property and depending on a given element of the moduli space men­tioned above. One aspect of this construction is that, as was also the case in [KLl]-[KL5], the underlying vector space of the tensor product module is not at all the tensor product vector space of the given modules. However, the theory does in fact provide an analogue of the concrete elements of or­dinary classical tensor product modules (the usual linear combinations of "tensors" of elements of the given modules), namely, the space of elements of the dual space of the tensor product vector space of two modules satisfy­ing a certain list of conditions, the most important of which is what we call the "compatibility condition," which is motivated by the Jacobi identity and which allows the abstract machinery to work. In fact, one of the main theorems in [HLl], [HL4] and [HL5] is that this space of vectors is in fact a module in a certain generalized sensej the proof of this result requires for­mal calculations based on the Jacobi identity. The desired tensor product module is then the contragredient module of this generalized module, in the case in which this generalized module is a module. This theorem enables us to establish the conceptual vertex-tensor-categorical properties of the tensor product operation, by analogy with the way in which one's ability to write down concrete tensor product vectors in a classical tensor product module enables one to establish the classical tensor-categorical properties (such as the associativity or commutativity properties). The machinery of both formal calculus and of the geometric interpretation of the notion of

354 Y.-Z. Huang and J. Lepowsky

vertex operator algebra enter heavily into these considerations. It should be emphasized that our theory applies (at present) to an ar­

bitrary "rational" vertex operator algebra satisfying a certain convergence condition and another natural condition, including the vertex operator al­gebras associated with the WZNW models (see [KZ], [TKJ, [FZ], [DL] for the necessary properties) and with minimal models (see [BPZ], [H4], [FZ], [Wa]), and especially including the moonshine module (see [FLM2], [Do]). Even though the moonshine module exhibits no monodromy, it is expected to possess rich vertex-tensor-categorical structure coming from the con­formal geometry. In particular, combined with some other structures (e.g., "vertex categories," which are analogues of categories, and "modular vertex tensor categories," which are higher-genus generalizations of vertex tensor categories) the present theory is expected to provide a resolution of the philosophical paradox described above and to shed light on such phenom­ena as "monstrous moonshine" (see [eN], [FLMI], [BI], [MN], [FLM2], [B2]). In the theory of tensor categories, we have the Tannaka-Krein recon­struction theorem (see for example [JS2]). For vertex tensor categories, we expect that a vertex tensor category together with certain additional struc­tures determines uniquely (up to isomorphism) a vertex operator algebra such that the vertex tensor category constructed from a suitable category of modules for it is equivalent (in the sense of vertex tensor categories) to the original vertex tensor category. This is supported by the fact that the vertex tensor categories contructed from the moonshine module V~ and from the trivial vertex operator algebra are not equivalent, even though the corresponding braided tensor categories are equivalent (and are both triv­ial). We also expect that this theory will give a better understanding of the conformal field theories constructed from representations of W-algebras. In addition, many of the notions, constructions and techniques will also apply to vertex operator algebras which are not rational and module categories which are quite general-not necessarily semisimple.

In the present paper we give a brief description of our theory. The complete theory will be given in a series of papers beginning with [HL4] and [HL5]. For the reader's convenience, we review basic concepts in the representation theory of vertex operator algebras in Section 2. For a certain element P( z) of the moduli space of spheres with three punctures and local coordinates, we describe the theory of P(z)-tensor products of two modules in Section 3. In Section 4, we introduce the notion of vertex tensor category, and we announce that the category of modules for a rational vertex operator algebra for which products of intertwining operators are convergent for variables in certain regions has the structure of a vertex tensor category. We also announce that any vertex tensor category gives rise to a braided tensor category.

Results and concepts in this paper were announced in talks presented

Vertex tensor categories 355

by both authors at the June, 1992 AMS-IMS-SIAM Joint Summer Re­search Conference on Conformal Field Theory, Topological Field Theory and Quantum Groups at Mount Holyoke College, and some of our work related to this material was presented by one of us (J. Lepowsky) at this Symposium in honor of Bertram Kostant. We are happy to dedicate this paper to Bert Kostant, teacher and friend, on the occasion of his sixty-fifth birthday.

Acknowledgments. We would like to thank D. Kazhdan, G. Lusztig and M. Finkelberg for interesting discussions, especially concerning the com­parison between their approach and ours in the special case of affine Lie algebras. We are also grateful to LM. Gelfand for initially directing our attention to the preprint of the paper [KL1] in his seminar at Rutgers Uni­versity and to O. Mathieu for illuminating comments on that preprint. We thank R. Borcherds for informing us that some years ago, he also began considering a notion of tensor product of modules for a vertex algebra. During the course of this work, Y.-Z. Huang has been supported in part by NSF grants DMS-8610730 (through the Institute for Advanced Study), DMS-9104519 and DMS-9301020 and J. Lepowsky by NSF grants DMS-8603151 and DMS-9111945 and the Rutgers University Faculty Academic Study Program. J. Lepowsky also thanks the Institute for Advanced Study for its hospitality.

2. Review of basic concepts

In this section, we review some basic definitions and concepts in the rep­resentation theory of vertex operator algebras. Everything in this section can be found in [FLM2] and [FHL], except for the following points: Here we use very slightly more general versions of the definitions of module and of intertwining operator than used in [FHLj, allowing complex rather than rational gradings for modules. We also consider the notions of "opposite vertex operator" and of "generalized module."

In this paper, all the variables x, xo, ... are independent commuting formal variables, and all expressions involving these variables are to be understood as formal Laurent series or, when explicitly so designated, as formal rational functions. (Later, we shall also use the symbol z, which will denote a complex number, not a formal variable.) We shall use the formal 8-function 8(x) = EnEZ xn and the convention that negative powers of a binomial are to be expanded in nonnegative powers of the second summand,

in expressions such as 8 ( X1::aX2 ), for example. See [FLM2] and [FHL] for

extensive discussions of the calculus of formal 8-functions, and see [HL4] for additional details concerning the subject of this paper.

356 Y.-Z. Huang and J. Lepowsky

We now quote the definition and basic "duality" properties of vertex operator algebras from [FLM2] or [FHL]:

Definition 2.1 A vertex operator algebra (over C) is a Z-graded vector space (graded by weights)

such that

v = II V(n)j for v E V(n), n = wt Vj nEZ

dim V(n) < 00 for nEZ,

V(n) = 0 for n sufficiently small,

equipped with a linear map V @ V - V[[x, X-l]], or equivalently,

V - (End V)[[x,x-lll

(2.1)

(2.2)

(2.3)

v t-+ Y(v,x) = Lvnx-n- l (where Vn E End V), (2.4) nEZ

Y (v, x) denoting the vertex operator associated with v, and equipped also with two distinguished homogeneous vectors 1 E V(o) (the vacuum) and w E V(2). The following conditions are assumed for u, v E V: the lower truncation condition holds:

un V = 0 for n sufficiently large (2.5)

(or equivalently, Y(u,x)v E V«x)))j

Y(1,x) = 1 (Ion the right being the identity operator)j (2.6)

the creation property holds:

Y(v,x)1 E V[[xll and lim Y(v,x)1 = v 0:-+0

(2.7)

(that is, Y (v, x) 1 involves only nonnegative integral powers of x and the constant term is v)j the Jacobi identity (the main axiom) holds:

(note that when each expression in (2.8) is applied to any element of V, the coefficient of each monomial in the formal variables is a finite sumj on

Vertex tensor categories 357

the right-hand side, the notation Y(',X2) is understood to be extended in the obvious way to V[[xo, xOI)]); the Virasoro algebra relations hold:

1 [L(m), L(n)] = (m - n)L(m + n) + 12 (m3 - m)6n +m ,oc (2.9)

for m, nEZ, where

and

L(n) = Wn+l for nEZ, i.e., Y(W, x) = 2: L(n)x-n- 2

nEZ

cE C;

L(O)v = nv = (wt v)v for n E Z and v E V(n);

d dxY(v,x) = Y(L(-1)v,x)

(the L ( -1 ) -derivative property).

(2.10)

(2.11) (2.12)

(2.13)

The vertex operator algebra just defined is denoted by (V, Y, 1,w) (or simply by V). The complex number c is called the central charge or rank of V. Homomorphisms of vertex operator algebras are defined in the obvious way.

Vertex operator algebras have important "rationality," "commutativ­ity" and "associativity" properties, collectively called "duality" properties. These properties can in fact be used as axioms replacing the Jacobi identity in the definition of vertex operator algebra, as we now recall.

In the propositions below, C[Xl,X2]S is the ring of rational functions obtained by inverting (localizing with respect to) the products of (zero or more) elements of the set S of nonzero homogeneous linear polynomials in Xl and X2' Also, ~12 (which might also be written as ~X1X2) is the operation of expanding an element of C[Xl, X2]S, that is, a polynomial in Xl and X2

divided by a product of homogeneous linear polynomials in Xl and X2, as a formal series containing at most finitely many negative powers of X2 (using binomial expansions for negative powers of linear polynomials involving both Xl and X2); similarly for ~21 and so on. (The distinction between rational functions and formal Laurent series is crucial.)

For any Z-graded, or more generally, (>graded, vector space W = 11 W(n) , we use the notation

W' = Ilwtn) (2.14)

for its graded dual. We denote the canonical pairing between W' and W by (-, .).

358 Y.-Z. Huang and J. Lepowsky

Proposition 2.2 (a) (rationality of products) For v, Vl, V2 E V and v' E V', the formal series (v', Y( vt, xt}Y( V2, X2)V) , which involves only finitely many negative powers of X2 and only finitely many positive powers of Xl, lies in the image of the map t12:

(2.15)

(2.16)

for some 9 E C[Xl, X2] and r, s, t E Z. (b) (commutativity) We also have

(v', Y(V2,X2)Y(Vt,Xl)V) = t2t!(Xl,X2)' (2.17)

Proposition 2.3 (a) (rationality of iterates) For v, Vl, V2 E V and v' E V', the formal series (V',Y(Y(Vt, XO)V2,X2)V) , which involves only finitely many negative powers of Xo and only finitely many positive powers of X2, lies in the image of the map £20:

(2.18)

where the (uniquely determined) element h E C[xo, X2]S is of the form

h( ) k(xo, X2) Xo, X2 = --'''-:':c......::,",--;'7

xoxi(xo + X2)t (2.19)

for some k E C[xo, X2] and r, s, t E Z. (b) The formal series (v', Y(vt, Xo + X2)Y(V2, X2)V), which involves

only finitely many negative powers of X2 and only finitely many positive powers of Xo, lies in the image of t02, and in fact

(2.20)

Proposition 2.4 (associativity) We have the following equality of ratio­nal functions:

t 1} (v', Y(Vl' Xt}Y(V2' X2)V) = (t:;l (v', Y(Y(vt, XO)V2, X2)V»lxo=XI_X2'

(2.21)

Proposition 2.5 In the presence of the other axioms, the Jacobi identity follows from the rationality of products and iterates, and commutativity and associativity. In particular, in the definition of vertex operator algebra, the Jacobi identity may be replaced by these properties.

Vertex tensor categories 359

We have the following notions of module and of intertwining operator for vertex operator algebras:

Definition 2.6 Given a vertex operator algebra (V, Y, l,w), a module for V (or V-module or representation space) is a IC-graded vector space (graded by weights)

such that

W = II W(n); for wE W(n), n = wt W; nEe

dim W(n) < 00 for n E IC,

W(n) = 0 for n whose real part is sufficiently small,

equipped with a linear map V ® W -+ W[[x,x- 1]], or equivalently,

V -+ (End W)[[x,x- 1]]

v I-t Y(v,x) = 2:vnX-n-1 (where Vn E End W) nEZ

(2.22)

(2.23)

(2.24)

(2.25)

(note that the sum is over Z, not IC), Y(v,x) denoting the vertex operator associated with v, such that "all the defining properties of a vertex operator algebra that make sense hold." That is, for u, v E V and W E W,

vn W = 0 for n sufficiently large (2.26)

(the lower truncation condition);

Y(l, z) = 1; (2.27)

(2.28)

(the Jacobi identity for operators on W); note that on the right-hand side, Y (11, xo) is the operator associated with V; the Virasoro algebra relations hold on W with scalar c equal to the central charge of V:

1 [L(m), L(n») = (m - n)L(m + n) + 12 (m3 - m)6m +n,oc

for m, nEZ, where

L(n) = Wn+l for nEZ, i.e., Yew, z) = 2: L(n)x-n- 2 ;

nEZ

(2.29)

(2.30)

360 Y.-Z. Huang and J. Lepowsky

L(O)w = nw = (wt w)w for nEe and wE W(n);

d dxY(v,x) = Y(L(-l)v,x),

(2.31)

(2.32)

where L( -1) is the operator on V.

We may denote the module just defined by (W, Y) (or simply by W). If necessary, we shall use Yw or similar notation to indicate that the vertex operators concerned act on W. Homomorphisms (or maps) of V-modules are defined in the obvious way. For V-modules WI and W2, we shall denote the space of module maps from WI to W2 by Homv(WI, W2).

For any vector space Wand any formal variable x, we use the notation

W{x} = {L anxnlan E W, nEe}. nEiC

(2.33)

In particular, we shall allow complex powers of our commuting formal vari­ables.

Definition 2.7 Let V be a vertex operator algebra and let (WI. YI), (W2' Y2) and (Wa, Ya) be three V-modules (not necessarily distinct, and possibly equal to V). An intertwining opemtor of type (W~tv2) is a linear map WI ® W2 -+ Wa{x}, or equivalently,

WI -+ (Hom(W2' Wa»{x}

w 1-+ Y(w, x) = L wnx-n- I (wn E Hom(W2' Wa» (2.34) nEiC

such that "all the defining properties of a module action that make sense hold." That is, for v E V, w(1) E WI and W(2) E W2, we have the lower truncation condition

(w(1»nW(2) = 0 for n whose real part is sufficiently large; (2.35)

the following Jacobi identity holds for the operators YI (v, .), Y2( v, .), Y3( v, .) and Y(', X2) acting on the element W(2):

(2.36)

Vertex tensor categories 361

(note that the first term on the left-hand side is algebraically meaningful because of the condition (2.35), and the other terms are meaningful by the usual properties of modules; also note that this Jacobi identity involves integral powers of Xo and Xl and complex powers of X2);

d dx Y (W(1),x) = Y(L(-I)w(l),X), (2.37)

where L( -1) is the operator acting on Wl.

The intertwining operators of the same type (W~~2) form a vector

space, which we denote by V~:W2. The dimension of this vector space is called the fusion rule for Wt, W2 and W3 and is denoted by N~:W2 ($ 00).

There are also duality properties for modules and intertwining opera­tors. See [FHL] and [DL] for details.

For any V-module (W, Y), we define the opposite vertex operator asso­

ciated to v E V by

Y*(v,x) = Y(eXL(1)(_x-2)L(O)v,x- l ), (2.38)

as in [HL4]. Then y* satisfies the following opposite Jacobi identity:

xo16 (Xl ~ X2 ) Y*(V2' X2)Y*(Vt, Xl)

-xo16 (X2 - Xl) Y*(vl,xdY*(V2' X2) -Xo

= x2l 6 ( Xl ~ Xo ) Y* (Y( Vl, XO)V2, X2) (2.39)

for Vl, V2 E V. The pair (W, Y*) should be thought of as a "right module" for V. We have Y** = Y (using formula (2.38) a second time to define Y**).

Let (W, Y), with

W = II W(n), (2.40) nEe

be a V-module, and consider its graded dual space W' (recall (2.14)). We define the contragredient vertex operators (called "adjoint vertex operators" in [FHLJ) Y'(v,x) (v E V) by means of the linear map

V -+ (End W')[[x, x- l Jl v I-> Y'(v,x) = L v~x-n-l (where v~ E End W'), (2.41)

nEZ

determined by the condition

(Y'(v,x)w',w) = (w', Y*(v,x)w) (2.42)

362 Y.-Z. Huang and J. Lepowsky

for v E V, w' E Wi, W E W. We give the space W' a C-grading by setting

W(n) = wtn) for n E C (2.43)

(cf. (2.14». The following theorem defines the V-module W' contmgredient to W (see [FHL], Theorem 5.2.1 and Proposition 5.3.1):

Theorem 2.8 The pair (W', yl) carries the structure of a V -module and (W", y,,) = (W, Y).

Given a module map"., : W l -+ W2 , there is a unique module map ".,' : W2 -+ W{, the adjoint map, such that

(2.44)

for W(I) E WI and W(2) E W2 • (Here the pairings (.,.) on the two sides refer to two different modules.) Note that

(2.45)

In the construction of the tensor product module of two modules for a vertex operator algebra, we shall need the following generalization of the notion of module recalled above:

Definition 2.9 A genemlized V -module is a C-graded vector space W equipped with a linear map of the form (2.25) satisfying all the axioms for a V-module except that the homogeneous subspaces need not be finite­dimensional and that they need not be zero even for n whose real part is sufficiently small; that is, we omit (2.23) and (2.24) from the definition.

3. The notion of P(z)-tensor product and two constructions

The symbol P(z) in the title of this section represents a geometric object associated with a fixed nonzero complex number z. Geometrically, to define a tensor product of modules for a vertex operator algebra, we need to spec­ify an element of the moduli space K of spheres with ordered punctures, the Oth puncture being negatively oriented and the others positively oriented, and with local coordinates vanishing at these punctures. More precisely, we need an element of the determinant line bundle over K raised to the power c, where c E C is the central charge of the vertex operator algebra. For detailed discussions of the moduli space K and its crucial role in the geometric interpretation of the notion of vertex operator algebra, see [HI], [H2], [H3] [H4]. In this paper, we give the notion and two constructions of tensor product associated with the particular element P(z) of K containing

Vertex tensor categories 363

CU{oo} with ordered punctures 00, z, 0 (00 being the (negatively oriented) Oth puncture and z and 0 the (positively oriented) first and second punc­tures, respectively) and standard local coordinates l/w, w-z, w, vanishing at 00, z, 0, respectively. This element P(z) is the geometric object corre­sponding to vertex operators or intertwining operators in the geometric interpretation of vertex operators and intertwining operators. The tensor products associated with other elements of K (or more precisely, elements of the determinant line bundle over K raised to the power c) can be defined analogously and can be constructed using the results described here for the constructions of the P(z)-tensor product.

Before giving the definition of P(z)-tensor product of two modules for a vertex operator algebra, we first recall the version of the definition of tensor product of two modules for a Lie algebra which will be our main guide. In the theory of Lie algebras we have the following standard notion of intertwining map (of type (~~2) among modules WI, W2, W3 for a Lie algebra V, with corresponding actions 11"1. 11"2, 11"3 of V: a linear map 1 from the tensor product vector space WI ® W2 to W3, satisfying the identity

for v E V, W(I) E Wb W(2) E W2. This "Jacobi identity for intertwining maps" agrees with the Jacobi identity for V when all three modules are the adjoint module. Let us call a product of WI and W2 a third module W3 equipped with an intertwining map 1 of type (w':'~J; we denote this by (W3, I). Then a tensor product of WI and W2 is a product (WI ® W2,®) such that given any product (W3, I), there exists a unique module map "l from WI ® W2 to W3 such that

I="lo®. (3.2)

Thus any tensor product of two given modules has the following property: The intertwining maps from the tensor product vector space of the two modules to a third module correspond naturally to the module maps from the tensor product module to the third module. Moreover, this universal property characterizes the tensor product module up to unique isomor­phism.

Note that this version of the definition of tensor product of two Lie algebra modules is based on knowledge of the intertwining maps relating three modules. The point is that in vertex operator algebra theory, we can formulate a notion of intertwining map (related to but distinct from the notion of intertwining operator), and using this we can define the appro­priate notion of tensor product of modules for a vertex operator algebra. This tensor product will not have the vector space tensor product as its underlying vector space. We proceed to describe how this works.

364 Y.-Z. Huang and J. Lepowsky

For any e-graded vector space W = 11 W(n) such that dim W(n) < 00

for each nEe, we use the notation

- II '* W= W(n) =W , (3.3) nEe

where as above' denotes the graded dual space and as usual * denotes the dual space of a vector space.

Let (V, Y, 1,w) be a vertex operator algebra and (W, Y) a V-module. It follows from the axioms that for any v E V and nEZ, there is a well­defined natural action of Vn on W. Moreover, for fixed v E V, any infinite linear combination of the Vn of the form En<N anvn (an E e) acts on W in a well-defined way.

Fix z E ex and let (WI, Yd, (W2' Y2) and (W3, Y3) be V-modules. By a P(z)-intertwining map of type (w,:",wJ we mean a linear map F WI ® W2 -t W 3 satisfying the condition

(3.4)

for v E V, W(1) E Wb W(2) E W2 (cf. the identity (3.1) and also the Jacobi identity (2.36)). The preceding paragraph shows that the left-hand side of (3.4) is well defined. We denote the vector space of P(z)-intertwining maps of type (W':"'W2) by M[P(z)1~~w2. A P(z)-product of WI and W2 is a V­

module (W3, Y3) equipped with a P(z)-intertwining map F of type (wwwJ We denote it by (W3, Y3; F) (or simply by (W3, F)). Let (W4' Y4; G) be another P(z)-product of WI and W2. A morphism from (W3, Y3; F) to (W4' Y4; G) is a module map 'fl from W3 to W4 such that

G=ijoF, (3.5)

where r; is the map from W 3 to W 4 uniquely extending 'fl. We define the notion of P(z)-tensor product using a universal property as follows:

Definition 3.1 A P(z)-tensor product of WI and W 2 is a P(z)-product

(WI \8IP(z) W 2, YP(z); \8IP(z»)

such that for any P(z)-product (W3, Y3; F), there is a unique morphism from

Vertex tensor categories 365

to (Wa, Ya; F). The V-module (WIl8lp (z) W2, Yp(z» is called a P(z)-tensor product module of WI and W2.

Remark 3.2 As in the case of tensor products of modules for a Lie algebra, it is clear from the definition that if a P(z)-tensor product of WI and W2 exists, then it is unique up to unique isomorphism.

The existence a P(z)-tensor product is not obvious. We shall discuss its existence and constructions under certain conditions, after first relating P(z)-intertwining maps to intertwining operators.

Let Y be an intertwining operator of type (w~~J. It follows from the definition of intertwining operator that for any complex number ( and any

W(I) E WI, Y(W(1),X)1 is a well-defined map from W 2 to Wa. xn=en" nEC

For brevity of notation, we shall write this map as Y( w(1), e'), but note that Y(w(1), e') depends on (, not on just e', as the notation might suggest. In this section we shall always choose log z so that

log z = log Izl + i arg z with 0:5 arg z < 271'. (3.6)

Arbitrary values of the log function will be denoted

lp(z) = log z + 2p7ri (3.7)

for p E Z. We now describe the precise connection between intertwining operators

and P(z)-intertwining maps of the same type. Fix an integer p. Let Y be an intertwining operator of type (w~~)' We have a linear map Fy,p :

WI 0 W 2 -+ Wa given by

(3.8)

for all W(I) E WI, W(2) E W2. Using the Jacobi identity for y, we see easily that Fy,p is a P(z)-intertwining map. Conversely, given a P(z)-intertwining map F, homogeneous elements w(1) E WI and W(2) E W 2 and nEe, we define (w(1»nW(2) to be the projection of the image ofw(1) 0W(2) under F to the homogeneous subspace of Wa of weight

wt W(I) - n - 1 + wt W(2)'

multiplied by e(n+I)lp(z). Using this, we define

YF,p(W(1),X)W(2) = 2:)w(1»nW(2)X-n- 1 ,

nEC

(3.9)

366 Y.-Z. Huang and J. Lepowsky

and by linearity, we obtain a linear map

W1 ®W2 ~ Wa{x}

W(l) ® W(2) ~ YF,p( W(I), x )W(2)

(recall the notation (2.33)). The proof of the following result is analogous to (and slightly shorter

than) the proof of the corresponding result for Q(z)-intertwining maps in [HL4J:

Proposition 3.3 For p E Il, the correspondence Y 1--7 Fy,p is a linear isomorphism from the vector space V~:W2 of intertwining operators of type

(W~W2) to the vector space M[P(Z)J~~W2 of P(z)-intertwining maps of

type (W~~2)' Its inverse map is given by F ~ YF,p'

The following immediate result relates module maps from a tensor product module with intertwining maps and intertwining operators:

Proposition 3.4 Suppose that WI I8Ip (z) W2 exists. We have a natural isomorphism

Homv(Wl I8Ip (z) W2 , Wa) ~ M[P(Z)J~~W2 'TJ 1--7 11 0 I8Ip (z)

and for p E Il, a natural isomorphism

Homv(WI I8Ip (z) W2, Wa)

'TJ

where Yfj,p = YF,p with F = 11 0 I8Ip (z)'

(3.10)

(3.11)

It is clear from Definition 3.1 that the tensor product operation dis­tributes over direct sums in the following sense:

Proposition 3.5 Let UI. ... , Uk, WI. ... , W, be V -modules and suppose that each Ui I8Ip (z) Wj exists. Then (lli Ui ) I8Ip (z) (llj Wj) exists and there is a natural isomorphism

(3.12)

Now consider V-modules WI. W2 and Wa. The natural evaluation map

WI ® W2 ® M[P(z)1!t~w2 ~ Wa

W(I) ® W(2) ® F 1--7 F(w(1) ® W(2») (3.13)

Vertex tensor categories 367

gives a natural map

(3.14)

Suppose that dim M[P{Z)]~~W2 < 00. Then (M[P{Z)]~~W2)* ® W3 is a V-module (with finite-dimensional weight spaces) in the obvious way, and the map .r~:W2 is clearly a P{z)-intertwining map, where we make the identification

(3.15)

This gives us a natural P{z)-product. Now we consider a special but important class of vertex operator alge­

bras satisfying certain finiteness and semisimplicity conditions.

Definition 3.6 A vertex operator algebra V is rational if it satisfies the following conditions:

1. There are only finitely many irreducible V-modules (up to equiva­lence).

2. Every V-module is completely reducible (and is in particular a finite direct sum of irreducible modules).

3. All the fusion rules (the dimensions of spaces of intertwining oper­ators) for V are finite (for triples of irreducible modules and hence arbitrary modules).

The next result shows that tensor products exist for the category of modules for a rational vertex operator algebra. It is proved by the same argument used to prove the analogous result in [HL4] for the Q{z)-tensor product. There is no need to assume that WI and W2 are irreducible in the formulation or proof, but by Proposition 3.5, the case in which WI and W2

are irreducible gives all the necessary information, and the tensor product is canonically described using only the spaces of intertwining maps among triples of irreducible modules.

Proposition 3.7 Let V be rational and let WI, W2 be V -modules. Then

exists, and in fact

k

W I i8JP (z) W2 = II (M[P{Z)]tt!W2)* ® Mi , (3.16) i=1

368 Y.-Z. Huang and J. Lepowsky

where {Ml, ... , Mk} is a set of representatives of the equivalence classes of irreducible V-modules, and the right-hand side of (3.16) is equipped with the V -module and P(z)-product structure indicated above. That is,

k

I8Ip (z) = L .r~:W2· (3.17) i=l

Remark 3.8 By combining Proposition 3.7 with Proposition 3.3, we can express Wl l8lp (z) W2 in terms of V~:W2 in place of M[P(Z)]~~W2.

The construction in Proposition 3.7 is tautological, and we view the argument as essentially an existence proof. We now give two constructions of a P(z)-tensor product (under suitable conditions).

For two V-modules (Wb Yd and (W2' Y2), we define an action of

V ® £+C[t, t- l , (Z-l - t)-l]

on (Wl ® W2)* (where as in [FLM2] and [HL4], £+ denotes the operation of expansion of a rational function of t in the direction of positive powers of t), that is, a linear map

(3.18)

by

(TP(Z) (X016 (XI:o- z) Yi(v,xd) A) (W(l) ® W(2»)

= z-16 (Xll z- xo) A(Yl(extL(1)(-XI2)L(0)v,xo)w(1) ® W(2»)

+xo16 (z -XI1) A(W(l) ® Y2*(V,Xl)W(2») (3.19) -Xo

for v E V, A E (Wl ® W2)*, w(1) E Wt, W(2) E W2, where

Yi(v, x) = v®x-16 (£). (3.20)

The formula (3.19) does indeed give a well-defined map of the type (3.18) (in gener-ating-function form); this definition is motivated by (3.4), in which we first replace Xl by x l l and then replace v byeXtL(1)(_xI2)L(0)v.

Let W3 be another V-module. The space V ® £+iC[t, rl, (Z-l - t)-l] acts on W3 in the obvious way, where v ® t n (v E V, n E Z) acts as the component Vn of Y(v, x). The following result, which follows immediately from the definitions (3.4) and (3.19), provides further motivation for the definition of our action on (Wl ® W2)*:

Vertex tensor categories 369

Proposition 3.9 Under the natural isomorphism

(3.21)

the maps in Hom(W~, (WI ® W 2 )") intertwining the two actions of

on W~ and (WI ® W2 )* correspond exactly to the P(z)-intertwining maps of type (W~W2).

Remark 3.10 Combining the last result with Proposition 3.3, we see that the maps in Hom(W~, (WI ® W2 )*) intertwining the two actions on W~ and (WI ® W2 )" also correspond exactly to the intertwining operators of type (w7~). In particular, for any intertwining operator Y of type (W~W2) and any integer p, the map Fy,p : W~ ---+ (WI ® W2 )* defined by

(3.22)

for w(3) E W~ (recall (3.8); the pairing (.,.) on the right-hand side is the obvious one) intertwines the actions of V ® L+qt, t-I , (z-I - t)-I] on W~ and (WI ® W2 )".

Write Yj,(z)(v,x) = Tp(z) (Yt (v, x»

(the specialization of (3.19) to V ® qt, t- I ]) and

Yj,(z)(W,x) = L L~(z)(n)x-n-2 nEZ

(3.23)

(3.24)

(reeall that w is the generator of the Virasoro algebra, for our vertex oper­ator algebra (V, Y, 1,w». We call the eigenspaces of the operator L~(z)(O) the weight subspaces or homogeneous subspaces of (WI ® W 2 )*, and we have the corresponding notions of weight vector (or homogeneous vector) and weight.

Suppose that G E Hom(W~, (WI ® W2 )*) intertwines the two actions as in Proposition 3.9. Then for w(3) E W~, G(w(3» satisfies the following nontrivial and subtle condition on ). E (Wl ® W2 )*: The formal Laurent series Yj,(z)(v,xo». involves only finitely many negative powers of Xo and

for all v E V. (3.25)

370 Y.-Z. Huang and J. Lepowsky

(Note that the two sides are not a priori equal for general A E (Wi ® W2)*.) We call this the compatibility condition on A E (Wi ® W 2)*, for the action Tp(z)·

Let W be a subspace of (Wi ® W2)*. We say that W is compatible for Tp(z) if every element of W satisfies the compatibility condition. Also, we say that W is (C-) graded if it is C-graded by its weight subspaces, and that W is a V-module (respectively, generalized module) if W is graded and is a module (respectively, generalized module) when equipped with this grading and with the action of YJ,(z)(·'x) (recall Definition 2.9). A sum of compatible modules or generalized modules is clearly a generalized module. The weight subspace of a subspace W with weight n E C will be denoted Wen) .

Given G as above, it is clear that G(W~) is a V-module since G in­tertwines the two actions of V ® C[t, t- i ]. We have in fact established that G(W~) is in addition a compatible V-module since G intertwines the full actions. Moreover, if H E Hom(W~, (Wi ® W2)*) intertwines the two actions of V ® C[t, t- i ], then H intertwines the two actions of V ® t+C[t, t- i , (z-i - t)-i] if and only if the V-module H(W~) is compat­ible.

Define

(3.26)

where Wp(z) is the set of all compatible modules for Tp(z) in (Wi ® W2)*. Then WilSip(z) W 2 is a compatible generalized module and coincides with the sum (or union) of the images G(W3) of modules W~ under the maps G as above. Moreover, for any V-module W3 and any map H : W~ ~ WilSip(z) W2 of generalized modules, H(W~) is compatible and hence H intertwines the two actions of V ® t+C[t, t- i , (Z-i - t)-i]. Thus we have:

Proposition 3.11 The subspace WilSip(z) W2 of (Wi ® W 2 )* is a gener­alized module with the following property: Given any V -module W3, there is a natural linear isomorphism determined by (3.21) between the space of all P(z)-intertwining maps of type (WWw) and the space of all maps of

, 1 2 generalized modules from W3 to WilSip(z) W2.

For rational vertex operator algebras, we have the following straight­forward result:

Proposition 3.12 Let V be a rational vertex operator algebra and Wi, W2 two V -modules. Then the generalized module WilSip(z) W2 is a module.

Now we assume that WilSip(z) W2 is a module (which occurs if V is ratio­nal, by the last proposition). In this case, we define a V-module W i l8l p (z) W2

Vertex tensor categories 371

by (3.27)

(observe the notation lSI' = ~!) and we write the corresponding action as YP(z). Applying Proposition 3.11 to the special module W3 = Wl ~P(z) W2 and the identity map W~ - WllSlp(z) W2, we obtain using (3.21) a canonical

P(z)-intertwining map of type (W1:V:mW2 ), which we denote

~P(z) : W l ® W2 - WI ~P(z) W2

W(l) ® W(2) 1-+ W(l) ~P(z) W(2).

It is easy to verify:

(3.28)

Proposition 3.13 The P(z)-product (WI ~P(z) W2, YP(z); ~P(z» is a P(z)-tensor product ofWl and W2.

More generally, dropping the assumption that WllSlp(z) W2 is a module, we have:

Proposition 3.14 The P(z)-tensor product of WI and W2 exists (and is given by (3.27}) if and only if WIlSlp(z) W2 is a module.

We observe that any element of WIlSlp(z) W2 is an element A of (WI ® W2 )" satisfying:

The compatibility condition (recall (3.25»

(a) The lower truncation condition: For all v E V, the formal Laurent series YJ,(z)(V,X)A involves only finitely many negative powers of x.

(b) The following formula holds:

for all v E V. (3.29)

The local grading-restriction condition

(a) The grading condition: A is a (finite) sum of weight vectors of (Wl ® W2)*.

(b) Let W>. be the smallest subspace of (WI ® W2)* containing A and stable under the component operators Tp(z)(V ® t n ) of the operators YJ,(z)(v,x) for v E V, n E Z. Then the weight spaces (W>')(n), nEe, of the (graded) space W>. have the properties

dim (W>.)(n) < 00 for nEe, (3.30)

(W>.)(n) = 0 for n whose real part is sufficiently small. (3.31)

372 Y.-Z. Huang and J. Lepowsky

We have the following result, which is quite straightforward:

Proposition 3.15 The action Yf,(z) of V ® Clt, t-1] on (W1 ® W2)* has the property

(3.32)

where 1 on the right-hand side is the identity map of (W1 ® W2)*, and the L( -I)-derivative property

d~ Yf,(z) (v, x) = Yf,(z)(L( -l)v, x) (3.33)

for v E V.

The following substantial result can be derived from the corresponding result for Q{z) in place of P(z) in [HL4]-[HL5]:

Theorem 3.16 Let A be an element of (W1 ® W2)* satisfying the compati­bility condition. Then when acting on A, the Jacobi identity for Yf,(z) holds, that is,

-1 (Xl - X2) '( )' ( ) Xo {j Xo Yp(z) U, Xl Yp(z) V, X2 A

foru,v E V.

-xC;l{j ( X2_~:1 ) Yf,(z) (v, X2)Yf,(z) (u, Xt}A

= X:;l{j ( Xl ~ Xo ) Yf,(z) {Y( u, Xo)v, X2)A

We also have:

(3.34)

Proposition 3.17 The subspace consisting of the elements of (W1 ® W2)* satisfying the compatibility condition is stable under the operators Tp(z){V® t n ) for v E V and nEZ, and similarly for the subspace consisting of the elements satisfying the local grading-restriction condition.

We can now formulate the main conclusions. We have another con­struction of W1iS1Q(z) W2:

Theorem 3.18 The subspace of (W1 ® W2)* consisting of the elements satisfying the compatibility condition and the local grading-restriction con­dition, equipped with Yf,(z) , is a generalized module and is equal to W 1iS1p(z) W2·

The following result follows immediately from Proposition 3.14, the theorem above and the definition of WlI81Q(z) W2:

Vertex tensor rotegories 373

Corollary 3.19 The P(z)-tensor product of WI and W2 exists if and only if the subspace of (WI ® W2)* consisting of the elements satisfying the com­patibility condition and the local gmding-restriction condition, equipped with YJ,(z) , is a module. In this case, this module coincides with the module W1iS1p(z) W2, and the contmgredient module of this module, equipped with the P(z)-intertwining map I8I p (z), is a P(z)-tensor product of WI and W2, equal to the structure (WI ~P(z) W2, YP(z); ~P(z») constructed above.

From this result and Propositions 3.12 and 3.13, we have:

Corollary 3.20 Let V be a mtional vertex opemtor algebm and WI, W2 two V -modules. Then the P(z)-tensor product (Wl l81p (z) W2, YP(z); ~P(z») may be constructed as described in Corollary 3.19.

4. Vertex tensor categories

In the preceding section, we have introduced the notion of P(z)-tensor product, exhibited some straightforward consequences and presented two constructions, the second of which is the important one. This theory of tensor products is in fact much richer than what we have seen in Section 3. In this section, instead of giving a complete description of the theory (which is the task of the series of papers mentioned in the introduction), we summarize the properties of tensor products of modules for a rational vertex operator algebra under the condition that products of intertwining operators converge in a suitable sense: We define the notion of vertex tensor category, and we announce that the category of modules for such an algebra gives the structure of a vertex tensor category and also that any vertex tensor category gives the structure of a braided tensor category.

To define the notion of vertex tensor category, we need the moduli space K and the determinant line bundle over K mentioned at the beginning of the preceding section. We also use the language of (partial) operads. For details, see [M], [Hl]-[H5] and [HL2]-[HL3]. As in these papers, k1 is the determinant line bundle over the moduli space K, and for any com­plex number c, k e is the determinant line bundle k1 raised to the power c. Then K, k1 and k e for any c E C are all CX-rescalable associative analytic partialoperads (see [HL2], [HL3], [H5]). We use the notations K(n), k 1(n) and keen) for n E N to denote the nth components of these operadic struc­tures (corresponding to spheres with n positively oriented punctures and one negatively oriented puncture). These components are path-connected. The substitution (composition) map for the partial operad k e is denoted by')'; this map describes the sewing operation. The identity element (of ke(1)) is denoted by i. Below we always use the convention that when we write an expression involving the substitution map of k e , we have assumed

374 Y.-Z. Huang and J. Lepowsky

that the expression exists. We also need the fiat section t/J : K _ [(c, given in [H2] and [H5], used to obtain the vertex operators and the central charge for the vertex operator algebra corresponding to a vertex associative algebra. We denote by J the element of K(O) containing the sphere CU{O} with the negatively oriented puncture 00 and the standard local coordinate vanishing at 00. Also recall the notation P(z) from the preceding section.

Definition 4.1 A vertex tensor (monoidal) category of central charge c E C consists of an abelian category V together with the following data (see below for the axioms):

1. For every Q E [(C(2), we have a bifunctor

~o: V x V- V,

called the Q-tensor product bifunctor.

2. For any Q E [(C(1), we have a functor

eo :V-V,

called the Q-quasi-identity functor, such that

where 1 is the identity functor from V to V.

3. We have an object V in V, called the unit object.

4. For any Qt, Q2, Q3, Q4 E [(C(2) satisfying

'Y(Ql;Q2,i) = 'Y(Q3;i,Q4),

we have a natural isomorphism

(4.1)

(4.2)

(4.3)

(4.4)

(where as above 1 is the identity functor from V to V), called the associativity isomorphism, so that in particular, for any objects Wt, W2, W3 in V we have an associativity isomorphism

5. For any Q E [(C(2), we have a natural isomorphism

(4.7)

Vertex tensor categories 375

called the commutativity isomorphism, where 0"12 denotes the non­trivial element of 82 and also the functor from V x V to itself given by the permutation 0"12, so that in particular, for any objects W 1, W2 in V we have a commutativity isomorphism

(4.8)

6. For any QI, Q2 E KC(2) and any homotopy class [r] of paths r from Q1 to Q2, we have a natural isomorphism

(4.9)

called the parallel transport isomorphism, so that in particular, for any objects W1 , W2 in V we have a parallel transport isomorphism

or

'Y{Q1; Rt} = R2,

'Y{R1; Q1.i) = R2,

'Y{R1; i, Qt} = R2

'Y{Q1;(2) = Q3.

(4.10)

(4.11)

(4.12)

(4.13)

(4.14)

Then we have a natural isomorphism

or

S[0]Qi!2 R- : eQ- 0 I8IR- -+ I8IR- , 1, 1 1 1 2

S[1]Ri!2 Q- : 18I.k 0 (eQ- x 1) -+ 18I.k , b 1 1 1 2

S[2]Ri!2 Q- : 18I.k 0 (1 x eQ- ) -+ I8IR-bIll 2

(4.15)

(4.16)

(4.17)

SQ93 Q- : eQ- 0 eQ- -+ eQ- , (4.18) 1, 2 1 2 3

respectively, called a substitution isomorphism, so that in particular, for any objects Wl. W2 in V we have the corresponding substitution isomorphism

or

respectively.

eQl (WI 18I.k1 W 2 ) -+ WII8I.k2 W 2 ,

eQl (W1) 18I.k1 W 2 -+ W1 18I.k2 W 2 ,

W1181.k1 eQl (W2 ) -+ WI 18I.k2 W 2

(4.19)

(4.20)

(4.21)

(4.22)

376 Y.-Z. Huang and J. Lepowsky

8. Let Ql E KC(l), Q2 E KC(2) and suppose that either

-y(Q2i 'IjJ(J) , i) = Ql (4.23)

or (4.24)

Then we have a natural isomorphism (the left unit isomorphism)

(4.25)

or (the right unit isomorphism)

(4.26)

respectively, where V ~Q2 . and· ~Q2 V are the functors from V to itself which take an object W to V~Q2 W and W~Q2 V, respectively, and a morphism f E Hom(Wl. W2 ) to Iv ~Q2 f and f ~Q2 lv, re­spectively, so that in particular, for any object W in V we have two unit isomorphisms

(4.27)

or (4.28)

respectively.

These data are required to satisfy the following axioms:

1. The symmetry condition

(4.29)

holds, where Q E KC(2) and [r Ql is the homotopy class of loops based

at Q homotopic to the loop r Q given by {'IjJ(P(e211"it» 10 :::; t :::; I}.

2. For any n E N, consider the directed graph On whose vertices are the decompositions of the elements of KC(n) into 'IjJ(J) and elements of KC(l) and KC(2) using the substitution map 'Y (the sewing oper­ation), and whose arrows are induced from the equality (4.4), from permutations of the orderings of the two positively oriented punctures of elements of KC(2), from homotopy classes of paths connecting el­ements of KC(2), from the equalities (4.11)-(4.14), and finally, from the equalities (4.23)-(4.24). We require the following: Let Dl and D2 be vertices of On and let Cl and C2 be chains of arrows from Dl to D2, giving us a diagram in On. We have a corresponding diagram whose

Vertex tensor categories 377

vertices are obtained by replacing the vertices of the original diagram by multifunctors constructed using the tensor product bifunctors, the quasi-identity functors, the unit object and the decompositions of el­ements of [(C(n) given by the original vertices, and whose arrows are the natural isomorphisms induced from the associativity, commuta­tivity, parallel transport, substitution and unit isomorphisms corre­sponding to the arrows in the original diagram. Then this diagram of multifunctors commutes, up to the natural automorphism of the multifuctor corresponding to D2 associated with the homotopy class of loops in [(C(n) obtained in the obvious way from the chains Cl and C2·

The following theorem is the main result of our theory:

Theorem 4.2 Let V be a rational vertex operator algebra of central charge c E C. Assume that for any intertwining operators Yi, i = 1, ... , n, of types (W~W;il+J, respectively, and any w(l) E WL W(2i) E W 2i , i = 1, ... , n, and W(2n+l) E W 2n+ 1,

is absolutely convergent when xf, ... , x~, m E C, are replaced by the com­plex numbers emlogzl, ... ,emlogzn (recall (3.6)) for any Zl,,,,,Zn E C satisfying IZll > ... > IZnl > O. Further assume that for any intertwin­ing operators Yl and Y2 of types (W':'~3) and (w':'wJ, respectively, any W(l) E W{ and W(S) E Ws, and any complex numbers Zl and Z2 satisfying IZll > IZ21 > IZl - z21 > 0,

(W(1),Yl(',Xl)Y2(',X2)W(S»)lxi=emIOgZl, x2'=emlogz2, mEC E (W2 ® W4)*

is the limit, in the locally convex topology defined by the pairing between (W2 ® W4)* and W 2 ® W4, of an absolutely convergent series of weight vectors in the module W 2ISlp(Zl-Z2) W 4 c (W2 ® W 4 )*. Then the category of V -modules has a natural structure of vertex tensor category of central charge c such that for each Z E CX , the tensor product bifunctor ~"'(P(z» associated to 1/J(P(z» E [(C(2) is equal to ~P(z).

It follows that for the familiar vertex operator algebras, we have:

Corollary 4.3 For the vertex operator algebra V associated with a WZNW model of positive integral level k, a minimal model of central charge 1 - 6 (p;;)2 (where p, q are relatively prime positive integers) or the moon­shine module, the category of V -modules has a natural structure of vertex

378 Y.-Z. Huang and J. Lepowsky

tensor category of central charge k'th (where d is the dimension of the finite­dimensional simple Lie algebra associated with the WZNW model and h is the dual Coxeter number), 1 - 6 (p;:)2 or 24, respectively.

Even in the case of the moonshine module, this vertex tensor category structure is nontrivial.

For tensor (monoidal) categories, there is a coherence theorem [ML]. For vertex tensor categories, we also have a coherence theorem, which re­duces the verification of the axiom 2 in the definition of vertex tensor cat­egory to the commutativity of certain simple diagrams.

We now describe the construction of a braided tensor (monoidal) cat­egory (see PSI]) from a vertex tensor category. Let V be a vertex tensor category. We first define the tensor product bifunctor to be

~ = ~t/J(P(l»' (4.30)

so that for two objects Wl and W2 in the underlying category of V, we have a tensor product

W 1 ~ W2 = W 1 ~t/J(P(l» W2 •

The associativity isomorphism a is defined by

rr t/J(P(2».t/J(P(1» ( ) ( ) a = .l[rlJ 0 At/J(P(l».t/J(P(l» : ~ 0 ~ x 1 -+ ~ 0 1 x ~ ,

(4.31)

(4.32)

where rl is a fixed path from 'I/J{P(2» to 'I/J{P{I» in kC(2) and where we view 7[r1) as a natural transformation from ~t/J(P(2» 0 (1 X ~t/J(P(l») to ~t/J(P(l» 0 (1 X ~t/J(P(l»)' The unit object is V. The unit isomorphisms are defined by

1 = C~(P(l» : V ~ . -+ 1, (4.33)

r = n!12(t{J(P(1))) 0 1ir 2) : . ~ V -+ 1, (4.34)

where r2 is a fixed path from 'I/J{P(1» to U12('I/J(P(1))). The braiding isomorphism is defined by

c = 1ir 3) 0 Ct/J(P(l» : ~ -+ ~ 0 U12, (4.35)

where r3 is a fixed path from U12('I/J(P(1))) to 'I/J(P(1».

Theorem 4.4 The underlying category of V, together with the tensor prod­uct ~ and the other data defined above, forms a braided tensor category. Different choices of [rl]' [r2] and [r3] give isomorphic braided tensor cate­gories.

In particular, we have:

Vertex tensor categories 379

Corollary 4.5 For the vertex operator algebra associated with a WZNW model, a minimal model or the moonshine module, the category of modules has a natural structure of braided tensor category.

These braided tensor categories yield the usual fusion rules and the usual fusing and braiding structures studied in conformal field theory (see [BPZ), [Ve) , [MS) , [BNS) , [Fi), etc.). For the moonshine module, this braided tensor category is trivial, although, as we have mentioned above, the vertex tensor category structure which produces it is nontrivial.

References

[BPZ) A. A. Belavin, A. M. Polyakov and A. B. Zamolodchikov, Infinite conformal symmetries in two-dimensional quantum field theory, Nucl. Phys. B241 (1984), 333-380.

[Bl) R. E. Borcherds, Vertex algebras, Kac-Moody algebras, and the Monster, Proc. Natl. Acad. Sci. USA 83 (1986), 3068-3071.

[B2] R. E. Borcherds, Monstrous moonshine and monstrous Lie super­algebras, Invent. Math. 109 (1992), 405-444.

[BNS] R. Brustein, Y. Ne'eman and S. Sternberg, Duality, crossing and Mac Lane's coherence, Isr. J. Math. 72 (1990), 19-37.

[CN) J. H. Conway and S. P. Norton, Monstrous moonshine, Bull. London Math. Soc. 11 (1979), 308-339.

[Do] C. Dong, Representations of the moonshine module vertex oper­ator algebra, in: Pmc. 1992 Joint Summer Research Conference on Conformal Field Theory, Topological Field Theory and Quan­tum Groups, Mount Holyoke, 1992, Contemporary Math., Amer. Math. Soc., Providence, to appear.

[DL) C. Dong and J. Lepowsky, Generalized Vertex Algebras and Rel­ative Vertex Operators, Progress in Math., Vol. 112, Birkhauser, Boston, 1993.

[DrI] V. Drinfeld, On quasi-cocommutative Hopf algebras, Algebra and Analysis 1 (1989), 30-46.

[Dr2) V. Drinfeld, On quasitriangular quasi-Hopf algebras and a certain group closely related to Gal(Q/Q), Algebra and Analysis 2 (1990), 149-181.

[Fa) G. Faltings, A proof for the Verlinde formula, to appear.

380 Y.-Z. Huang and J. Lepowsky

[Fi] M. Finkelberg, Fusion categories, Ph.D. thesis, Harvard University, 1993.

[FHL] I. B. Frenkel, Y.-Z. Huang and J. Lepowsky, On axiomatic ap­proaches to vertex operator algebras and modules, preprint, 1989; Memoirs Amer. Math. Soc. 104, Number 494, 1993.

[FLMl] I. B. Frenkel, J. Lepowsky and A. Meurman, A natural represen­tation of the Fischer-Griess Monster with the modular function J as character, Proc. Natl. Acad. Sci. USA 81 (1984), 3256-3260.

[FLM2] I. B. Frenkel, J. Lepowsky and A. Meurman, Vertex Operator Alge­bras and the Monster, Pure and Appl. Math., Vol. 134, Academic Press, Boston, 1988.

[FZ] I. B. Frenkel and Y. Zhu, Vertex operator algebras associated to representations of affine and Virasoro algebras, Duke Math. J. 66 (1992), 123-168.

[FS] D. Friedan and S. Shenker, The analytic geometry of two­dimensional conformal field theory, Nucl. Phys. B281 (1987),509-545.

[Go] D. Gorenstein, Finite Simple Groups. An Introduction to Their Classification, Plenum Press, New York, 1982.

[Gr] R. L. Griess, Jr., The Friendly Giant, Invent. Math. 69 (1982), 1-102.

[HI] Y.-Z. Huang, On the geometric interpretation of vertex operator algebras, Ph.D. thesis, Rutgers University, 1990; Operads and the geometric interpretation of vertex operator algebras, I, to appear.

[H2] Y.-Z. Huang, Geometric interpretation of vertex operator algebras, Pmc. Natl. Acad. Sci. USA 88 (1991), 9964-9968.

[H3] Y.-Z. Huang, Applications of the geometric interpretation of ver­tex operator algebras, in: Proc. 20th International Conference on Differential Geometric Methods in Theoretical Physics, New York, 1991, ed. S. Catto and A. Rocha, World Scientific, Singapore, 1992, Vol. 1, 333-343.

[H4] Y.-Z. Huang, Vertex operator algebras and conformal field theory, Inti. J. Mod. Phys. A7 (1992), 2109-2151.

[H5] Y.-Z. Huang, Operads and the geometric interpretation of vertex operator algebras, II, in preparation.

Vertex tensor categories 381

[HLl] Y.-Z. Huang and J. Lepowsky, Toward a theory of tensor prod­ucts for representations of a vertex operator algebra, in: Proc. 20th International Conference on Differential Geometric Methods in Theoretical Physics, New York, 1991, ed. S. Catto and A. Rocha, World Scientific, Singapore, 1992, Vol. 1,344-354.

[HL2] Y.-Z. Huang and J. Lepowsky, Vertex operator algebras and op­erads, in: The Gelfand Mathematical Seminars, 1990-1992, ed. L. Corwin, 1. Gelfand and J. Lepowsky, Birkhauser, Boston, 1993, 145-161.

[HL3] Y.-Z. Huang and J. Lepowsky, Operadic formulation of the no­tion of vertex operator algebra, in: Proc. 1992 Joint Summer Re­search Conference on Conformal Field Theory, Topological Field Theory and Quantum Groups, Mount Holyoke, 1992, Contempo­rary Math., Amer. Math. Soc., Providence, to appear.

[HL4] Y.-Z. Huang and J. Lepowsky, A theory of tensor products for module categories for a vertex operator algebra, I, to appear.

[HL5] Y.-Z. Huang and J. Lepowsky, A theory of tensor products for module categories for a vertex operator algebra, II, to appear.

[J] V. F. R. Jones, Hecke algebra representations of braid groups and link polynomials, Ann. Math. 126 (1987), 335-388.

[JSl] A. Joyal and R. Street, Braided monoidal categories, Macquarie Mathematics Reports, Macquarie University, Australia, 1986.

[JS2] A. Joyal and R. Street, An introduction to Tannaka duality and quantum groups, in: Category theory, Como, 1990, ed. A. Carboni, M.C. Pedicchio and G. Rosolini, Lecture Notes in Math. 1488, Springer-Verlag, Berlin, 1991,413-492.

[KLl] D. Kazhdan and G. Lusztig, Affine Lie algebras and quantum groups, International Mathematics Research Notices (in Duke Math. J.) 2 (1991), 21-29.

[KL2] D. Kazhdan and G. Lusztig, Tensor structures arising from affine Lie algebras, I, J. Amer. Math. Soc. 6 (1993), 905-947.

[KL3] D. Kazhdan and G. Lusztig, Tensor structures arising from affine Lie algebras, II, J. Amer. Math. Soc. 6 (1993), 949-1011.

[KL4] D. Kazhdan and G. Lusztig, Tensor structures arising from affine Lie algebras, III, J. Amer. Math. Soc. 7 (1994), 335-381.

382 Y.-Z. Huang and J. Lepowsky

[KL5) D. Kazhdan and G. Lusztig, Tensor structures arising from affine Lie algebras, IV, J. Amer. Math. Soc. 7 (1994), 383-453.

[KZ) V. G. Knizhnik and A. B. Zamolodchikov, Current algebra and Wess-Zumino models in two dimensions, Nucl. Phys. B247 (1984), 83-103.

[Kl) T. Kohno, Linear representations of braid groups and classical Yang-Baxter equations, in: Braids, Santa Cruz, 1986, Contem­porary Math. 78 (1988), 339-363.

[K2) T. Kohno, Monodromy representations of braid groups and Yang­Baxter equations, Ann. Inst. Fourier 37 (1987), 139-160.

[Ll) J. Lepowsky, Perspectives on vertex operators and the Monster, in: Proc. 1987 Symposium on the Mathematical Heritage of Hermann Weyl, Duke Univ., Proc. Symp. Pure Math., Amer. Math. Soc. 48 (1988), 181-197.

[L2] J. Lepowsky, Remarks on vertex operator algebras and moonshine, in: Proc. 20th International Conference on Differential Geometric Methods in Theoretical Physics, New York, 1991, ed. S. Catto and A. Rocha, World Scientific, Singapore, 1992, Vol. 1, 362-370.

[LW] J. Lepowsky and R.L. Wilson, A new family of algebras underlying the Rogers-Ramanujan identities and generalizations, Proc. Natl. Acad. Sci. USA 78 (1981), 7254-7258.

[ML] S. Mac Lane, Categories for the Working Mathematician, Graduate Texts in Math., Vol. 5, Springer-Verlag, New York, 1971.

[MN] G. Mason, Finite groups and modular functions (with an appendix by S. P. Norton), in: Representations of Finite Groups, Proc. 1986 Summer Research Institute, Arcata, ed. P. Fong, Proc. Symp. Pure Math., Amer. Math. Soc. 47 (1987), 181-210.

[M] J. P. May, The geometry of iterated loop spaces, Lecture Notes in Math. 271, Springer-Verlag, 1972.

[MS) G. Moore and N. Seiberg, Classical and quantum conformal field theory, Comm. Math. Phys. 123 (1989), 177-254.

[RT) N. Reshetikhin and V. G. Turaev, Invariants of 3-manifolds via link polynomials and quantum groups, Invent. Math. 103 (1991), 547-597.

Vertex tensor categories 383

[SV] V. V. Schechtman and A. N. Varchenko, Arrangements of hyper­planes and Lie algebra homology, Invent. Math. 106 (1991), 139-194.

[SI] J. D. Stasheff, Homotopy associativity of H-spaces, I, funs. Amer. Math. Soc. 108 (1963), 275-292.

[S2] J. D. Stasheff, Homotopy associativity of H-spaces, II, funs. Amer. Math. Soc. 108 (1963), 293-312.

[TK] A. Tsuchiya and Y. Kanie, Vertex operators in conformal field theory on ]pi and monodromy representations of braid group, in: Conformal Field Theory and Solvable Lattice Models, Advanced Studies in Pure Math., Vol. 16, Kinokuniya Company Ltd., Tokyo, 1988, 297-372.

[TUY] A. Tsuchiya, K. Ueno and Y. Yamada, Conformal field theory on universal family of stable curves with gauge symmetries, in: Advanced Studies in Pure Math., Vol. 19, Kinokuniya Company Ltd., Tokyo, 1989,459-565.

[Va] A. N. Varchenko, Hypergeometric functions and representation theory of Lie algebras and quantum groups, to appear.

[Ve] E. Verlinde, Fusion rules and modular transformations in 2D con­formal field theory, Nucl. Phys. B300 (1988), 360-376.

(Wa] W. Wang, Rationality of Virasoro vertex operator algebras, In­ternational Mathematics Research Notices (in Duke Math. J.) 7 (1993), 197-211.

[Wi1] E. Witten, Non-abelian bosonization in two dimensions, Comm. Math. Phys. 92 (1984), 455-472.

[Wi2] E. Witten, Quantum field theory and the Jones polynomial, Comm. Math. Phys. 121 (1989), 351-399.

Y.-Z. Huang, Department of Mathematics, University of Pennsylvania, Philadelphia, PA 19104 Current address: Department of Mathematics, Rutgers University, New Brunswick, NJ 08903

J. Lepowsky, Department of Mathematics, Rutgers University, New Brunswick, NJ 08903

Received January 21, 1994

Enveloping Algebras: Problems Old and New

Anthony Joseph

Dedicated to Professor Bertram Kostant on his sixty-fifth birthday

o. Introduction

I recently came across a list of twenty problems on enveloping algebras that I had presented during a meeting on Enveloping Algebras in Oberwolfach during August 1982. This was a time when the study of primitive ideals had reached a particularly interesting stage and before quantum groups had been invented. Of these roughly half were settled and need not be mentioned further. On the remaining problems relatively little progress ha.<; been made despite the availability of new geometric information par­ticularly on orbital varieties and of our experience with quantum groups. In the following sections some of these and some new questions are reviewed. Throughout 9 denotes a semisimple Lie algebra and U(g) its enveloping algebra.

1. The composition factors of primitive quotients

1.1. Let P be a primitive ideal of U(g) and consider the quotient U(g)/P as a U(g) bimodule. This is a Harish-Chandra module for the pair (g x g, G) where G denotes the adjoint group corresponding to the diagonal copy of g. In particular U(g)/P has finite length. Thus a problem is to determine the composition factors of U(g)/P with their multiplicities. Let Pmin be the minimal primitive ideal contained in P. There is a simple recipe due to Duflo [Du] to describe the composition factors of P / P min j but this does not give their mUltiplicities. Consequently it does not even give the composition factors of U(g)/ P. A related problem is to determine the multiplicities of U(g)/P as a G module.

When P is a so-called minimal primitive ideal then a complete solution to the second problem obtains from a result of Kostant [K1] and states that the multiplicity of any simple G module is exactly the dimension of its zero weight space. This can be interpreted through the surjectivity of U(g) onto the G locally finite part of the space of linear Verma module maps [CD, 5.5] through Frobenius reciprocity. It was a key to the representation

386 A. Joseph

theory of 9 and of course to the study of primitive ideals. For regular characters this carries over [CD, 2.1.2, 6.3] to induced ideals (relative to an arbitrary parabolic subalgebra) and even to certain natural generalizations of them [GJ, 5.3]. For P maximal one has three separate results; but all are very limited. First for regular characters there is the result of [GJ, 5.3] mentioned above. Secondly for "unitary characters" , which are rather special, one has a result of Barbasch and Vogan [BV, Thm. IV] which is extended in (J12] and in [Ba 1]. Thirdly one can determine the multiplicity of the adjoint representation (J7] but even this becomes extremely delicate for singular characters.

In the first problem difficulties involving singular characters are avoided and one can expect a more elegant solution. For P minimal it is settled by the solution to the Kazhdan-Lusztig conjectures [BB, BK] and to a problem of Dixmier (J2, BG, GJ]. Moreover a complete solution can be given for an induced ideal and its natural generalizations (J3, 4.8]. Finally U(g)fP has a simple socle as first noted by Duflo [Du, Lemma 6] and this can be completely determined (J3, 4.9; I, 1.3].

A promising approach to these two problems involves a variation on the Jantzen sum formula. We reproduce below in a slightly shortened form an unpublished note (J9] on this question.

1.2. Let 9 = n+ E9 ~ E9 n- be a triangular decomposition and let <p : U(g) -t

U(~) be the corresponding Harish-Chandra map [D, 7.4]. We shall often write n+ simply as n. Let K denote the space of harmonic elements of U(g) defined by Kostant [K1] and as reproduced in [D, 8.1.4], viewed as a 9 module for the adjoint action. Fix an isomorphism class E of a simple finite dimensional 9 module. Set n = dim Eo and fix a basis {Xi}i=l for Eo. Recall [K1; D, 8.3.9] that E occurs with multiplicity n in K. Fix n linearly independent copies of E in K and choose Xij E K so that {Xij }i=l forms the basis of Eo for the lh copy of E in K. Let <PE denote the matrix with entries {<p(Xij)}i,j=l ' The determinants det <PE were first computed by Parthasarathy, Ranga Roo and Varadarajan [PRV] and further generalized by Kostant [K2]. However the original proof was rather long. Here we give a short proof based on reasoning in (J6, 5.4] and Hesselink's formula [H]. Notably this carries over to the quantum case (JL3, 3.6] and this gives a key result on Verma module annihilators (see 3.2, Theorem 5).

Fix oX E ~*. Let M(oX} denote the Verma module with highest weight oX and L(oX} its unique simple quotient. Recall [K1; D, 8.4.3] that K complements Annu(g)M(oX} in U(g).

Lemma 1. (i) E'L(oX) = 0 ¢=} <p(E')(oX} = 0 for any 9 submodule E' of U(g).

Enveloping Algebras: Problems Old and New 387

(ii) rk<pE()..) = [U(g)/Annu(g)L()") : E]. In particular det <PE()..) =I- 0 if M()..) ~ L()..).

(iii) ).. is a zero of det <PE of orner ~ [AnnKL()") : E].

Proof. For (i) let e~ denote a choice of highest weight vector of M()") and M()")- the ~ stable complement of Ce~ in M()..). Then <p(E')()..) = o {=> E'eA C M()")-. Yet E'M()..) = E'U(n-)e~ = U(n-)E'e~ and so E'e~ C M()")- means that E'M()..) is a strict submodule of M()"), equivalently that E'L()..) = 0 as required.

Since K + Annu(g)L()") :) K + Annu(g)M()") = U(g) we have U(g)/Annu(g)L()") = K/AnnKL()..). Then (ii) follows from (i). Further­more [K/AnnKL()..) : E] = [K : E] - [AnnKL()..) : E] and so by (ii) we can choose the copies of E in K so that [AnnKL()") : E] columns of <PE are zero. This proves (iii). I

Since M()") ~ L()..) on a Zariski dense set [D, 7.6.23] we conclude from Lemma 1 that det <PE is a non-zero polynomial on ~*. This is computed below.

Let R denote the set of non-zero roots and R+ the choice of positive roots defined by the triangular decomposition above. Let p denote the half sum of the positive roots. For each 0: E R+ let HOI. denote the corresponding coroot. For each n E N+ let EnOl. denote the no: weight space of the finite dimensional simple 9 module E. Given w E W (the Weyl group) set w.).. = w().. + p) - p.

Lemma 2. For each 0: E R+ and each n E N+, the polynomial (HOI. + p(Ha) - n)dim Ena divides det <PE.

Proof. Set An,a = {).. E ~* I ()..+p)(Ha) = n, ()..+ p)(H{3) ~N+, V f3 E

R+ \ {o: n. Take).. E An,a. By [D, 7.6.23] M()..) has length 2 with M(sa.)") its unique simple submodule. Set J = AnnKL()..). Then J =

{a E K I aM()") C M(sa.)")}. We sketch below a well-known argument to

prove that [J: E] = dim EnOl.. Let t denote the diagonal copy of 9 in 9 x g. Given U(g) modules

M, N consider Homc(M, N) as a U(g x g) module as in [GJ, 1.51. Set

L(M,N) = {a E Homc(M,N) I dim U(t)a < oo}. Via [Kl; D, 8.3.9, 8.4.3]

the action of U(g) on M()") defines an isomorphism K ~L(M()"), M()..». Under this isomorphism J identifies with L(M()"), M(sa.)"» which by [GJ, 1.9] identifies with the principal series module of t finite elements of (M(sa.)") ® M()..»*. By Frobenius reciprocity we conclude that [J: E] = dim E~-8a.~ = dim EnOl..

388 A. Joseph

Set ~~ = {jL E ~·I (jL + p)(HoJ = OJ. Expand det !.pE as a polynomial

in Ha + p(Ha) - n over ~~ . Take A E An•a. By (iii) of Lemma 1 and the above it follows that A is a zero of det !.p E of order ~ dim Ena. Since A can be made to have an arbitrary value on ~~ we conclude that the coefficient of (Ha + p(Ha) - n)i in det !.pE vanishes if i < dim Ena. Thus (Ha + p(ha) - n)dim En .. divides det !.pE, as required. •

Proposition. Up to a non-zero scalar

det !.pE = II II (Ha + p(Ha) - n)dim En ..

nEN+ aER+

Proof. Since U(~) = S(I) and is a unique factorization domain, we conclude from Lemma 2 that the right hand side divides the left hand side. The proof may then be concluded by showing that det !.p E has at most the degree of the right hand side above.

Let E(jL) denote the unique simple finite dimensional module with highest weight jL. Let dm(jL) denote the number of times that E(jL) occurs in the space of homogeneous harmonic elements of U(g) of degree m. Set P,.. (q) = EmEN dm (jL )qm. Since the Harish-Chandra map !.p can only lower degree, it follows that

degdet !.pE ~ L mdm(jL) = P~(I) . mEN

Let lew) denote the reduced length of w. By Hesselink [H, Thm.] it follows that P,..(q) is just the coefficient of eO in the expression

Using prime to denote the derivative with respect to q we obtain

and so

Q~(I) = L (I :-ea_ a ) Q,..(I) = L L e-nach E(jL) .

aER+ nEN+ aER+

Enveloping Algebras: Problems Old and New 389

We conclude that

P~(1) = L L dim E(J.t)nOt . nEN+ OtER+

Combined with our previous observations this provides the required esti­mate on degrees and so proves the proposition. •

1.3. Fix A E ~ .. and recall [Ja, 5.1] how the Jantzen filtration M(A) = MO(A) ::> Ml(A) ::> ••• , of M(A) is defined. Set U = U(g) and extend the base field C by an indeterminate t. It is easy to check that the Verma module M(A + tp) is simple over U(g) ®c C(t) and hence admits a non­degenerate contravariant form with values in C(t). Let C[t]o denote the localization of C[t] at < t >. Since A + tp is also a linear map from ~ to C[t]o we may also define M(A + tp) as a module over (j := U ®c C[t]o admitting a contravariant form <, > with values in C[t]o. For each i E N set

Mi(A+tp) = {m E M(A+tp) 1< m,M(A+tp) > c < t i >} .

This is clearly a decreasing filtration by g := 9 ®c C(t)o modules. Set Mi(A) = Mi(A + tp) ®C[tJo C[t]o/ < t >. Then Mi(A) is a decreasing filtration of M(A) by 9 modules, hence stationary since M(A) has finite length. Again nMi(A + tp) = 0, by the simplicity property noted above and so Mi(A) = 0 for i sufficiently large.

In [Ja, 5.3] Jantzen established a sum rule for the Jordan-Holder factors in Mi (A). Here we should like to establish a corresponding sum rule for annihilators Ji(A) := Annu(M(A)/Mi(A». Unfortunately we are not quite able to do this. Instead we consider the ideals

where U is identified with (j / < t >.

Lemma 1. Ii (A) C Ji (A), for all i with equality when i = 1.

Proof. Straightforward. • A characterization of the ideals Ii(A) comes from the following. Given

any submodule M of M(A + tp) let MJ.' denote its J.t weight space.

Lemma 2. If M rt Mi+l(A + tp), then M>'+tp rt< ti+l > e>'+tp'

390 A. Joseph

Proof. Suppose M ¢. MHl(>'+tp). Then there exists m E M such that < m, M(>' + tp) >¢.< t H1 >. Without loss of generality we can assume m to be a weight vector say of weight /L and we can assume /L maximal with this property. If /L = >. + tp we can write m = ae,Htp : a E C(tlo. Then the above relation gives a '/.< tH1 > and so M)'+tp ¢.< tH1 > e).+tp, as required. Now suppose /L t >. + tp. We can find a weight vector n E M(>. + tp) of weight /L such that < m, n >'/.< tH1 >. Let Xa E 9 denote the element of the Chevalley basis (defining <, » corresponding to the root a. Since M(>. + tp) is generated over U(n-) by e).+tp we have /L < >. + tp and we can write n = Ea>o x_anp +a' Then < m, n >= Ea>o < xam,np +a >c< tH1 > by the choice of /L. This contradiction proves the lemma. •

Now assume>. + p is regular. Set R). = {a E R I (av,>.) E Z}. Let W). denote the subgroup of W generated by the reflections Sa : a E

R).. Set Rt = R). n R+. Then there is a unique w E W). such that /L + P := w(>. + p) is dominant. Assume a E R).. Then>. - Sa . >. is a positive integer multiple of a if and only if wa > O. Moreover in this case /L - Swa·/L = «wa)V, /L + p)wa = (aV , >. + p)wa = w(>. - sa.>'). Finally we note that a standard computation using that M(/L) is projective in the 0 category gives

[L(M(/L), M(swa./L» : E] = dim Ep-swo.p

for each simple finite dimensional U(g) module E.

Proposition. One has

00

i=l

= [L(M(/L), M(swa./L» : EJ

Proof. The result obtains by calculating the largest power of t dividing det rpE(>' + tp) in two different ways. First by 1.2, Proposition it takes the

Enveloping Algebras: Problems Old and New

value

dim Ena nEJIi aER+ l(aV ,oX+p)=n

= L dim E oX- 8a •oX

aERt l(aV ,oX+p»O

= L [L(M(Jl),M(swa.Jl»: E], by (*) . aERtlwQERt

391

Second one may suppose M8(A) = 0 for some s E N+. Then J8(A) = AnnuM(A) which is generated by its intersection with the centre Z of U. Take Z E Z. Then zeHtp = <p(Z)(A + tp)eoX+tp and so Z - <p(Z)(A + tp) E

Ann{jM(A + tp). It easily follows that Ji(A) ::> Ann{jM(A + tp) ®{j U ::> AnnuM(A) = J8(A). In particular JB(A) = J8(A).

It remains to show that the largest power of t dividing det <PE(A + tp) is just

8

L[Ji(A)/JB(A) : E] i=l

Let E' be a 9 stable subspace of U. We can write

for some i. As in (i) of 1.2, Lemma 1 this gives E'M(A + tp) = < t i > eHtp mod M-(A + tp). In particular (E'M(A + tp)hHp = < t i > eHtp and so by Lemma 2 one has E'M(A + tp) c Mi(A + tp), that is E' c Ann{j(M(A + tp)/Mi(A + tp».

Let M denote the set of dim Eo x dim Eo matrices with entries in C[t]o. We can write <p(xij)eoX+tp = mijeHtp for some (mij) EM. Since C[t]o is a principal ideal domain we can find a, b E M invertible such that amb is diagonal and with entries that can be assumed to be of form til. Then up to units det <PE(A + tp) = det m = det(amb) = IT til, so the required power of t is just Eli. On the other hand we can view the X~j := E aiuxuvbvj E K ®cC[t]o as new basis elements for the subspaces Eo in the dim Eo copies of E occurring in K ®cC[t]o. By our first computation, the copy of E corresponding to the basis {x~j} 1!:~ Eo of Eo exactly lies in Ann{j(M(A + tp)/Mti(A + tp». We conclude that [Ji(A)/J8(A) : E] = #{j I lj ~ i}. Hence

8 8

L[Ji(A)/JB(A): E] = L#{j I lj ~ i} = Llj , i=l i=l

392 A. Joseph

as required. • Recall that Jip.) :,) Ann M(A), V i. It follows that the Ji (A)/Ji+l(A)

are sub quotients of U(g)/Ann M(A) viewed as a U(g) bimodule. The sim­ple factors of U(g)/Ann M(A) are precisely [see GJ, 1.12 for example] the V( -Y./l, -/l) := L(M(/l), L(Y./l» : Y E WA • We can therefore define the Jordan-Holder multiplicities [Ji (A)/Ji+1(A) : V(-Y./l,-/l)]. We remark that M(A)/M1(A) = L(A) and so J1(A) = Ann L(A) by Lemma 1. Again [GJ, 1.16]

[L(M(/l),M(Swa./l» : V( -Y·/l, -/l)]

= [M(swa./l) : L(Y·/l)]

= p s .. ""v(l)

where Px,v(q) denotes the Kazhdan-Lusztig polynomial for the pair x, Y E

W A•

One can ask if our sum rule holds with E replaced by V( -Y·/l, -/l), that is does one have

00

i=1

V Y E WA, where /l = W.A is dominant? It is clear that (**) implies the proposition. Yet t isomorphism classes (t-types) do not distinguish the V( -y.A, -A), so the converse may fail. However we note that by trans­lation principles the expression [Ji (W- 1./l)/Ji+1(W-1./l): V(-Y./l,-/l)] is independent of the choice of /l+P in the set (/lo+P(R»++ of dominant and regular elements of flo + P(R), where P(R) denotes the lattice of integral weights. Again by contravariance of the Shapovalev form (c.f. (Ja, 1.6]) the above expression is invariant under replacing Y by y-1.

Now [see GJ, 1.12 for example] the isomorphism class E(/l - Y./l) with extreme weight /l - Y./l is the so-called unique minimal t-type of V( -Y./l, -/l). Two such t types E(/l - Y./l) and E(/l - z./l) : y, Z E W A,

coincide exactly if x(/l-Y./l) = /l-Z./l for some x E W. By the above trans­lation principles we can assume this holds for all /l E (/lo+P(R»++ with a possible dependence of x on fl. However since IWI < 00 we can assume that x does not depend on /l at least on some dense subset A C (1-'0 + P(R»++ and then by linearity in I-' on the whole of ~". We are therefore reduced to examining when the relation x(l - Y)(I-' + p) = (1- z)(1-' + p) V I-' E ~*

Enveloping Algebras: Problems Old and New 393

implies either y = z or taking account of the last remark in the previous paragraph, that y = Z-l.

Corollary. Suppose 9 is simple of type A n - l . Then (**) holds for all A regular with J.t := W.A dominant.

Proof. Under the hypothesis W identifies with the set of permutations of the set {I, 2"", n}. Without loss of generality we can replace ~ .. by the linear span of the ei : i = 1,2"", n where aei = ea(i), V a E W. Set a = xy in the relation x(l - y) = 1 - z. Then x + z = 1 + a. It follows that {I, 2, ... ,n} is a disjoint union of two subsets X,.1 such that x(i) = a(i), z(i) = i, ViE X and z(j) = a(j), x(j) = j, V j E.1. From this we conclude that a = xz. Consequently y = z as required. •

The above analysis fails in say type On.

1.4. The basic idea that we wish to express here is that it should be possible to read off the multiplicities occurring in the left hand side of 1.3( **) by a comparison of weight filtrations of appropriate principal series modules. For this there is no essential loss of generality in assuming that we are in the integral case and more particularly that J.t = o.

Given e, '1/ E ~ .. with e - '1/ E P(R), let L( -e, -'1/) denote the principal series module of t locally finite elements of (M(e) ® M('I/»*' Here we note that the latter identifies with L(M('I/) , 8M(e» where 8 denotes the 0-duality functor (cf. [GJ, 1.8]). Its simple factors are amongst those of the form V(-e',-'l/') := L(M('I/'),L(e'» with '1/' E w.'I/ dominant, e' E w.e· In particular, L( -x.O, -y.O) : x, yEW has simple factors amongst the V( -z.O, -0) : z E W.

Let 1t denote the extended Hecke algebra for W which we recall is t t

generated over IC[q2", q-2"] by elements Tw : W E W satisfying the standard Hecke algebra relations as defined in say [L1, 3.3]. Now let D~ : yEW be Lusztig's basis for 1t as defined in [L1, 5.1]. Then we can write Ty-tTx in the form

where Nx,y,z(q) is a Laurent polynomial in q! . Fix J.t E P(R) dominant. Let Wo denote the unique longest ele-

ment in W and set J.t- = wo.J.t. For example J.t- = -2p, if J.t = O. Though we have not attempted to check this, the methods of fCC, Sect. 4] should imply that the coefficient of q-i/2 in Nx,y,z(q) deter­mines the multiplicity of V( -z.J.t-, -J.t-) in the ith gradation step for the

394 A. Joseph

weight filtration of L(-x.J.t_, -Y.J.t-). Now recalling that 8M(J.t_) S!

M(J.t-) identify L( -J.t-, -J.t-) with L(M(J.t_), M(J.t-» and hence with U(g)/Ann M(J.t-). Recall that we have a (Kunze-Stein) intertwining oper­ator from L(-J.t-, -J.t-) to L(-x.J.t_, -x.J.t-) whose image [Du, 3, Prop.lO] is just U(g)/Ann L(x.J.t-). Up to an obvious shift it should respect the weight filtration; but we do not need to know this for the present discussion. Set A = x.J.t-. This suggests that the simple factors of U(g)/Ann L(A) should just be those simple factors which have the same gradation level with respect to the weight filtration of L( -11--, -J.t-) compared to that of L(-x.I1--,-x.J.t-). More generally the simple factors of Ji (A)/Ji+l(A) should be just those which have their gradation degree increased by 2i.

A first difficulty with the above suggestion is that because of multiplici­ties in the Jordan-Holder series for U(g)/Ann M(A) it is not unambiguous. However it does determine the left hand side of 1.3(**). Specifically we are saying that the left hand side of 1.3(**) should equal the difference be­tween the sum of the exponents in the polynomials N""""z(q) and Ne,e,z(q), namely N~,,,,,z(l) - N~,e,Al). That this does in fact equal the right hand side of 1.3(**) is expressed by the following purely combinatorial

Lemma. For all x, yEW one has

N~,,,,,y(l) - N~,e,y(l) = Ps.,,,,,y(l) aER+I",aER-

Proof. Let B C R+ be the set of simple roots. Assume x E W \ {e} and choose a E B such that x, := SaX < x. Then

T",-lT", = T",'_lT;",T""

= (q -l)T",'_lTs",T"" + qT",'_lT"" .

We conclude that

Hence

Now T",-lT"" specializes to x-1sax at q = 1 and so we obtain from [Ll, 5.1.7]

Enveloping Algebras: Problems Old and New 395

Substitution in the above gives the recurrence relation

Now let x = sal Sa2 ... Sak : O:i E B be a reduced decomposition and set x~ = sal Sa2 ... Sa. , Xi = SaHI SaH2 ••• sak' Then by (**) we obtain

k

N~,,,,,y(l) = ~ P",-:-IS",.",. (1) + N~,e,y(l) . ~ .. i=l

It remains to observe that f3i := xi10:i E R+, that xf3i = X~O:i E R­and finally that {Pi: i = 1,2, .. ·, k} = {o: E R+ I XO: E R-}. Of course this last fact is quite standard. The lemma is proved. •

The above suggestion indicates that there is a weight filtration on each I i(>.)/Ii+1(>.) induced by the weight filtrations of L(-J.L-, -J.L-) and L( -x.J.L-, -x.J.L-). Roughly speaking this is obtained as a piece of the former which has been shifted upwards by 2i levels in the latter. Combi­natorically, this should be described by polynomials in two indeterminates t, s. More precisely we require for each pair x, yEW a Laurent polynomial M""y(t,s) such that

and

{ Ne,e,y(q) : t2 = q, S = 1

M""y(t,s) = N""""y(q) : t2 = s2 = q

00

M""y(l,s) = Ls2i [Ii (x.J.L_)/Ii+l(x.J.L_): V(-Y.J.L-,-J.L-)]. i=O

Finally the induced weight gradation {(Ii(X.J.L_)/Ii+l(x.J.L_))j}jEZ{; is given by M""y(t, s) through the formula

M""y(t,s) = Ltjs2i[(Ii(X.J.L_)/Ii+l(x.J.L_))j+2i: V(-Y.J.L-,-J.L-)]. i,j

In the following we propose a formula for M""y(t, s) based on induction on the reduced length f(x) of x.

Set Me,y(t, s) = Ne,e,y(t2) and assume that M"",y(t, s) is given when x' = SaX < x, for some 0: simple. Set L(x, y) := L( -x.J.L-, -Y.J.L-). Recall that we have a four step exact sequence (J4, 3.3]

0--+ L(x,x') --+ L(x',x') --+ L(x,x) --+ L(x,x') --+ O.

396 A. Joseph

Correspondingly (*) gives

In the light of the above exact sequence this just means that L(x, x') viewed as a weighted submodule of L(x', x') is shifted upwards by two levels to reappear as a weighted quotient of L(x, x). The difficulty is to identify the terms in L(x', x') which correspond to L(x,x'). Now from the relation M"",y(t, t) = N""''''',y(q) it follows that we can attach to the simple factor V( -y./1--, -/1--) not just a monomial in t (which determines its place in the weight filtration of L(x', x'» but more precisely a monomial in t, s where the exponent in s is just the amount by which this factor is shifted from its place in the weight filtration of L(e, e). We conjecture that the correct identification of L(x, x') corresponds to those factors having the highest possible powers of s or equivalently as having been shifted to the greatest

possible extent from their places in L(e, e). Then M""y(s, t) is obtained from M"",y(s,t) by multiplying each such monomial by s2.

Unfortunately, we do not know if M""y(s, t) so defined is independent of the reduced decomposition of x. For this we would seem to need a genuine two-variable Hecke algebra.

One of the motivations for the above proposal comes from the com­parison of the weight filtrations of L(e, e) and L(wo,wo). Thus from [LI, 5.1.7] one checks that

In particular,

so

T", = l·("') E P""y(q)q-t(y)/2 D~ yEW

I = E Pe,y(q)q-t(y)/2 D~ yEW

N (t2 ) = p. (t2 )r£CY) • e,e,y e,Y

Now apply the Kazhdan-Lusztig involution u of1l defined by u(Tw) = T;!u u(q) = q-l to (***) recalling [LI, 5.1.8] that D~ = q-tCwo)/2C",woTwo and u(Cy) = Cy, 'V yEW by definition of the Kazhdan-Lusztig C basis of 1l. This gives

T2 = qtCwo) ~ P. (q-l)qtCY)/2 D' Wo ~ e,y y ,

yEW

Enveloping Algebras: Problems Old and New 397

so

We conclude that L(wo,wo) has the reversed filtration to L(e,e). Our proposal is just the most obvious way to achieve this reversal.

2. The quantization of orbital varieties

2.1. Let G be Lie group with Lie algebra g. After Kirillov [Ki] the tangent space to each point in a G orbit of g* admits a G-invariant non-degenerate alternating bilinear form which turns out to define a closed two-form. In geometric terms as emphasized by Kostant the orbit is a G equivariant symplectic manifold and the "orbit method" developed notably by Kirillov (see Kirillov's articles in [AD and Kostant [K3] attempts to assign to each such structure a unitary representation of G. If the symplectic form is not exact the construction requires a line bundle satisfying certain integrality conditions (prequantization) introduced and developed by Kostant [K3] and furthermore it is necessary to eliminate half the variables (polarization) as introduced by Kirillov [Ki] and developed notably by Auslander and Kostant [AK]. These constraints were understood by physicists (notably J.-M. Souriau - see (L), (Z) of [SoD in terms of quantization rules and the selection of say position co-ordinates.

The above considerations lead to the following construction. Given f E, g* find a subgroup P of G containing StabGf such that dim G IP = ! dim GIStabGf and ([p,p],!) = 0 where p = Lie P. The last condition means that x.-. f(x) defines a one dimensional U(p) module Cf and so one may form the induced module M(f) := U(g) ®U(p) C f . The first condition is <lquivalent to p being a maximal isotropic subspace with respect to the bilinear form (x, y) .-. ([x, y],!) on 9 x 9 or equivalently that P f is a Lagrangian subvariety of the orbit Gf. It is intended to ensure that M(f) is simple or at least has a primitive annihilator.

Rather than M(':\), Kirillov had considered a suitable subspace of func­tions on the line bundle GXpCf (where f may be required to satisfy certain conditions imposed by unitarity). For 9 semisimple, this corresponds in al­gebraic terms to taking the submodule of M(f)* of locally finite vectors with respect to a "maximal compact" subalgebra t of g. This last con­struction is functorial and the corresponding higher derived functors (due to Zuckerman) lead to further interesting modules. Moreover middle co­homology admits a pairing from which it makes sense to demand unitarity [V, Chaps. 5,6]. Detailed information on the history of the orbit method

398 A. Joseph

may be obtained from the preface of this volume. Concerning its imple­mentation in quantum groups some history and results can be found in [JU, AA.1-5].

2.2. In general a subgroup P having the above properties fails to exist and even when it does one may fail to obtain all the interesting modules in this fashion. Nevertheless one may still attempt to assign to each "good" Lagrangian subvariety of a co-adjoint orbit a "good" U(g) module. Here a "good" U(g) module can be taken to be holonomicj but this might be too weak. Yet unitary would be too strong for algebraists. If the modules are to be highest weight modules (which generalize the induced case above) then the Lagrangian subvarieties should be the so-called orbital varieties. The latter are defined as follows. Identify g* with 9 through the Killing form. Recall (1.2) the triangular decomposition of 9 and let B be the subgroup of G with Lie algebra b := n EB~. Orbits meeting n are called nilpotent. An orbital variety is defined to be an irreducible component of the locally closed set 0 n n for some nilpotent orbit O. From work of Spaltenstein and Steinberg one may show [J5, Sects. 7, 9] that every orbital variety is Lagrangian, that every irreducible Lagrangian subvariety of 0 n n is orbital and that the map w ~ B(n nnw) is a surjection of W onto the set of closures of orbital varieties. Let I(w) denote the ideal of S(n-) of functions vanishing on B(n nnw) en. Then S(n-)/I(w) identifies with the set of regular functions on the closure of the unique orbital variety V(w) dense in B(nnnW ). (It is not true that V(w) = B(nnnW ).)

2.3. Already the geometry of orbital varieties has several open and chal­lenging problems. The fibres of the map w ~ V(w), called geometric cells, are known for sl(n) but not in general. A. Melnikov [Ml,3] has made some important progress on the much more difficult problem of determining the inclusion relations V(w') c V(w) which is unsolved even for sl(n). There is however a conjectural solution to this question based [J5, 9.9] on some unknown quantities involving Euler numbers. The characteristic polynomi­als Pw which measure the growth rate of weight subspaces of S(n-)/I(w) introduced in [J5, Sect. 2] have the remarkable properties of separating the I(w) and defining Springer's representations [Ho] of W. They can also be calculated from these Euler numbers [J8, R]. Very little is known about the I(w). It is false [M3,T] that V(w) is a union of orbital varieties outside possibly sl(n).

2.4. A rather loose interpretation of the program of quantizing or­bital varieties is to find for each w E W a simple highest weight mod­ule L(A) whose associated variety is V(w). This amounts to showing that ..jgr Annu(n-)eA = I(w). A difficulty is that the left hand side need not

Enveloping Algebras: Problems Old and New 399

be a prime ideal, nor is it obvious what it would be in this case. These difficulties are resolved if 9 is of type An by a result of A. Melnikov [M2] showing the left hand side to be prime and by say [J5, 8.14]. Consequently this question is settled in type An but not yet in general.

2.5. A rather more precise quantization program consists of giving S(n-)/I(w) a U(g) module structure. Assuming that the ~ module struc­ture is preserved (at least up to an overall translation of weights) this means finding a highest weight module N(>"), not necessarily simple, whose high­est weight vector eA satisfies gr Annu(n-)eA = I(w). Here the choice of >.. is much more delicate than in 2.4. When V(w) is the nilradical of a parabolic subalgebra., then N(>") can be chosen to be an induced module and this corresponds to the set-up in 2.1. One of the first non-trivial cases for which this fails is when 9 = s(4) and V(w) is the subvariety in n van­ishing on e21, e43 and the "corner" minor e31 e42 - e41 e32. The quantization of this orbital variety was actually first studied by Kostant [K4] and gave rise to a unitary highest weight module. Actually it turns out that ev­ery unitary highest weight module has the property that Annu(m-)e>. is a prime ideal [J1O] where m- is the (commutative) subalgebra of 9 spanned by the non-compact negative root vectors (e3b e4b e32, e42 in the above case). Moreover every orbital variety lying in m can be quantized in the above sense [HJ] and the resulting unitary modules belongs to the so-called Wallach set. Unitary modules outside this set correspond to inducing from non-linear characters of t and passing to a quotient of the induced module. Of course m is the nilradical of a parabolic subalgebra and is commutative and this is a rather rare phenomenon.

Even for 9 = sl(6) the above program can fail if one demands N(>") to be simple. However in each of the two bad cases which arose E. Ben-1010 [B1,2] has shown that one may find (paradoxically) a non-simple N(>") having the required property. Even in these cases it was a quite delicate matter to verify that certain natural generators of I(w) formed a complete set and further that they could all be found in gr Annu(n-)e>.. Of course here>.. does not determine N(>") and so its actual value has less signifi­cance. Indeed it already happens that the unique simple quotient L(>") of that particular N(>") is also a quantization of a (different) orbital variety. Though it was not specifically checked I believe there was only one choice of N(>") in each of the above two examples.

2.6. An attractive proposal whose origin lay in the work of Goncharov [G] and was further pursued in [LSS] and then notably by Levasseur and Stafford [LS] is to directly study the algebra of differential operators Vw on the closure V(w) of an orbital variety and to hope that (by some mir­acle) this admits a quotient of U(g) as a subalgebra. Remarkably this

400 A. Joseph

does work for those orbital varieties associated to unitary modules (2.5) and moreover if V (w) is not the nilradical of a parabolic subalgebra then the map U(g) -- Vw is surjective [LS, JlO]. On the other hand if Yew) is the unique 3 dimensional orbital variety in type G2 , whilst Yew) ad­mits a "natural" U(g) module structure one fails to have an embedding U(g)jAnn Yew) ~ V w , the latter being a result of Zahid [Z].

2.7. Hopefully one may get further insight into these quantization prob­lems using quantum groups. Thus the generators of lew) in S(n-) are often minors which one has to modify slightly to obtain generators for U(n-). Due to Drinfeld duality the distinction between U(n-) and S(n-) is effaced on passage to quantum groups and indeed there is just one object Uq(n-) which admits (different) specializations «(JL2, Sect. 6], and Section 4 be­low) giving both U(n-) and S(n-). In some sense either set of generators should lift to quantum minors.

3. Annihilators of Verma modules for quantized enveloping algebras

3.1. Let Uq(g), or simply U, denote the "simply-connected" Drinfeld­Jimbo quantized enveloping algebra of g. (This was introduced indepen­dently in [DKP] and (JL2] , the latter paper being received by the editors in May 1991. Many of the results noted below would fail for the original Drinfeld-Jimbo algebra. For more details on this see (JL2, 3].) It is a C(q)­algebra with generators Xo" Y-a : a E B, r(A) : A E peR) satisfying the relations

r(A)r(JL) = r(A + JL),

r(A)Xar ( -A) = q~A,aV)Xa, r(A)Y_ar ( -A) = q;;(A,aV)Y_a

xaY-{3 - Y-{3xa = Oa,{3 (r(a) - r~~a») qa - qa

(qa = q(a,a)/2, Oa,{3 - Kronecker delta) together with the quantum Serre relations. Let U+ (resp. U-) denote the subalgebra generated by the Xa (resp. Y-a): a E B and set UO = C(q)r(P(R». Then the analogue of triangular decomposition (1.2) is that the multiplication map gives an isomorphism U- ® UO ® U+ ~ U of vector spaces. A character A on UO defines a one dimensional module C(q)A for V+ := UO ® U+ on which the augmentation ut of U+ acts by zero. By analogy with the classical (enveloping algebra) case one calls the induced module M(A) := U ®v+

C(q)A' a Verma module.

Em/eloping Algebras: Problems Old and New 401

3.2. Since Uq(g) is also a Hopf algebra it admits an adjoint action. Unlike U(g) this action is not locally finite. However the subalgebra F(Uq(g», or simply F(U), of Uq(g) formed from ad U locally finite vectors is quite large. Moreover it has the following particularly elegant description [JL2, 4.10). Set P+(R) = {A E P(R)I(A, oY) EN, V 0: E B}.

Theorem 1. One has

F(U) = EB (ad U)r( -2A) . .xEP+(R)

In the above each (ad U)r( -2A) is isomorphic to End L(A) (which is finite dimensional) as a U module. In particular the identity on L(A) maps to an element Z.x E Z(U), where Z(U) denotes the centre of U. Consequently

Theorem 2. One has

Z(U) = EB C{q)z.x . .xEP+(R)

In particular Z(U) is a polynomial algebra on generators Z"'i : i = 1,2"" ,t.

Drinfeld [Dr, Sect. 3) gave a quite different general construction for central elements in a Hopf algebra admitting an R matrix. This was orig­inally applied to a slightly bigger algebra than U; but can be used to recover Theorem 2. The analogue of Theorem 2 for the original Drinfeld­Jimbo algebra was first indicated by Rosso [Ro) but who did not quite determine the centre, and then independently by DeConcini and Kac [DK) and [JL1). The above proof is a slightly more sophisticated version of [JL1). The centre of U for not too small odd roots of unity is calculated in [DKP).

Theorem 1 has a natural generalization PH, 7.1.6) to the quantized enveloping algebra of an arbitrary Kac-Moody algebra g. Via PH, 7.1.1) one similarly obtains central elements; but only in the 0 completion Uf(g) which nevertheless still act on any Verma module (for a precise definition, see for example PH, 3.3.8)). Drinfeld's R matrix method also applies here.

Although (ad U)r( -2A) is not a subalgebra of U one has PL4, 9.2]

Theorem 3. The action of U on L(A) induces a U module isomorphism (ad U)r(-2A)~End L(A).

Kostant's separation of variables theorem extends to F(U) and indeed there is an ad U stable subspace K of F(U) satisfying [JL2, 5.4, 7.4] the

402 A. Joseph

Theorem 4. (i) F(U) = Z(U) ® K.

(ii) [K: L('\)] = dim L('\)o.

An important question in the representation theory of Uq(g), which leads for example to a quantum version [JL3] of DuBo's description [Du] of Prim U(g), is to compute AnnuM(A). Of course Z(U) acts by scalars on M(A), so one should first show that

Theorem 5. For every character A on UO one has

AnnK M(A) = 0 .

The proof which is surprisingly difficult [JL3] requires apart from The­orem 4 a quantum analogue of 1.2, Proposition. A question which arose in the analysis is how do the central elements arising in (ad U)r( -2,\), via the decomposition in (i) of Theorem 4 act on a given M(A). In the case of z>. an explicit formula can be given [JL1, Jll, 7.1.19]. Set Q(R) = 'llB. Although from general principles z>. must vanish on some M(A) for some A defined over the algebraic closure C(q), of C(q), it does not vanish on any A of the form ql-' : J.t E Q(R), that is when r('\)(ql-') := q(>',I-') , V ..\ E P(R). One may ask if this holds for the remaining central elements, more precisely does one have

Ann M(ql-') n (ad U)r( -2,\) = 0

for all J.t E Q(R), ,\ E P(R)+ ?

3.3. Theorem 5 does not quite show that AnnuM(A) is generated by its intersection with Z(U) and indeed this is a tricky question. Relative to F(U) the structure of U can be described as follows. Obviously T< :=

r( -2P(R)+) is an Ore subset of F(U) and so J(U) .- T<l F(U) is a subalgebra of U. One has [Jll, 7.1.13] the

Proposition. As a J(U) module, U is free on generators r(..\) where the ..\ run over a system of representatives of P(R)/2P(R).

It is clear that AnnJ(u)M(A) is generated by its intersection with Z(U). On the other hand one may easily check that each J(U)r('\) is ad U stable. If"\ f/. 2P(R) then by Theorem 1 it admits no locally finite vectors. This will lead to the

Corollary. For every character A on UO one has AnnuM(A) U Annz(u)M(A).

Enveloping Algebras: Problems Old and New 403

Proof. Set I = Annz(u)M(A) which is an ideal of codimension 1 in Z(U). By Theorem 2 one has a vector space isomorphism Z(U) = Z(U)/I E9 [.

Fix A E P(R) \ 2P(R) and set J' = J(U)r(A). By Theorem 4 (i), J' is a free Z(U) module. Thus one obtains an isomorphism J' = J' / J' I E9 J'I of ad U modules. Consequently J' / J' [ admits no finite dimensional ad U submodules.

Fix representatives 0 = AO, At. A2,·· ., An E P(R) for P(R)/2P(R) and set J i = J(U)r(Ai), Ki = Jd JJ. Identify Ko with T<K. Then

n

[J := U/U Annz(U)M(A) ~ E9 Ki i=O

is an isomorphism of ad U modules. It suffices to show that M(A) is a faithful [J module.

Suppose 0 f. U E AnnuM(A) when written as a sum U = E Ui : Ui E K i , has a minimum number m of components. It follows from Theorem 5 that m > 1. Multiplying by a suitable r(A) : A E -P(R)+ we can assume that Uo f. 0 and Uo E K without loss of generality. Take i > 0 such that Ui =1= o.

Set V = (ad U)u c AnnuM(A). Suppose v E V \ {O}. By the minimality of the number of components of U it follows that Vo f. 0 and Vi f. o. Furthermore the map Vo t--+ Vi of Vo := (ad U)uo to Vi := (ad U)Ui is well-defined and a non-zero homomorphism of ad U modules. Recalling that K is a locally finite ad U module this is impossible by the first part.

I

4. Canonical root vectors for quantized enveloping algebras

4.1. In [Br] R.K. Brylinski gave an interesting interpretation of Hesselink's formula for generalized exponents given by Q/-,(q) of 1.2 above. This goes as follows. Let e be a regular nilpotent element chosen so that in the corresponding 5(2) triple (e, h, f) one has that h is a dominant element of ~. It is convenient for what follows to take e = EaEB ea , where the ea

form part of a Chevalley basis for g. This is possible by say [D, 8.1.1]. One may assume that (ea , e-a ) = 1, Va E R+ where ( , ) denotes the Killing form. Now fix J1 E P(R)+ and let E(J1) denote the corresponding simple finite dimensional module with highest weight J1. For each weight subspace E(J1)v of E(J1) and each mEN, set

404 A. Joseph

Then one may define a q-character for E(J.t) through

00

chqE(J.t) = L L dim(E(J.t):!, /E(J.t):!'-l)qme", m=OvEf)"

which of course reduces to the ordinary character for q = 1. It would be pleasant to suppose that

However this fails. One difficulty is that the coefficients of qm in QIl(q) are not all positive. However Brylinski [Br, 6.4] showed that if one restricts to weights v which are dominant and regular then the formula that (*) predicts for dim E(J.t)'Z' / E(J.t):!'-l is correct. For more recent developments see [Bro].

4.2. Let us consider the much easier question of determining the q­

character of the" dual 6M(J.t) of the Verma module with highest weight J.t. Here the action of the positive root vectors on 6M(J.t) comes from the action of the negative root vectors on M(J.t). Since M(J.t) is just U(n-) as a U(n-) module this action is independent of J.t. Consequently apart from a factor of ell the q-character of 6M(J.t) is independent of J.t. Precisely

On the other hand there is an embedding E(J.t) ~ 6M(J.t) which has the further property that for all v E NB one has E(J.t)Il-v~ 6M(J.t)Il-v for J.t sufficiently large. It follows easily from Brylinski's result and (*) that

chq6M(O) = II (1 - qe-a)-l . aER+

We start with a simple independent proof of this fact. Our original intention had been to recover Brylinski's result by using the dual form of the standard resolution of E(J.t) in terms of Verma modules; but we were unable to make this work.

4.3. Through the isomorphisms

where the last map is symmetrization, the action of U(g) on M(J.t) defines a map U(g) --+ End 8(n-) whose image IC, Sect. 5] lies in the algebra

Enveloping Algebras: Problems Old and New 405

of differential operators on n-. Moreover under Fourier transform the el­ements of 9 become first order differential operators with the zero order term carrying the dependence on J-t. In particular 9 acts by derivations on oM(O) and these can be viewed as vector fields on the flag variety G/B. Consequently 4.2 (**) has the following interpretation. Namely there ex­ist distinguished weight vectors a_a E S(n-) defined by the conditions ea_a =1= 0, e2a_a = o. Moreover each such weight vector occurs if and only if a is a positive root and then with multiplicity one. Finally the a_a: a E R+ generate S(n-) as a polynomial algebra. Our aim is to recover this result in a manner that will apply to the quantized enveloping algebra Uq(g) thereby obtaining a canonical choice of root vectors in U­(or in U+). Here it is convenient to assume 9 simple. This entails no real loss of generality.

4.4. Identifying M(J-t) with U(n-) as in 4.3 means that the action of U(n-) is just left multiplication. This commutes with right multiplication. Translated to S(n-) we obtain two mutually commuting actions of n+ on S(n-) by derivations. Now assume that 9 is simple and let {3 denote the unique highest root. According to 4.3 there exists a unique up to scalars element a_{3 E S(n-) such that e2a_{3 = o. Applying the derivations e~ : a E R+ corresponding to the second action of n+ one obtains further weight vectors e~a_{3 satisfying e2(e~a_{3) = O. Let us indicate briefly why this gives exactly all the a_a : a E R+ described in 4.3 via the conclusion of 4.2 (**).

The adjoint action of n on b transposes to an action of n on b*. Identify b* with b- = n- E9 ~ through the Killing form. This gives a linear action of non b- with ~ the subspace of n invariant elements. Then n acts on S(b-) and hence on S(b-)/ < (ha - 6»: a E B > ~ S(n-) by derivations.

\O',O'}

Let e~ denote the derivation corresponding to a E R+.

Lemma 1. The set ofn invariant elements of S(n-) reduces to scalars.

Proof. Lift the order relation J-t ?: v on peR) defined by J-t - v E WB, to a total order. Then the matrix e~ (e_{3) : a, {3 E R+ is upper triangular with non-zero scalars on the diagonal. An easy argument (cf. [JI, 2.6]) completes the proof. •

Remark. Suppose instead one divides by the ideal generated by the ha : a E B then a similar argument shows the well-known fact that S(n-)n is generated bye_a : a E B.

The action of n+ on S(n-) coming from M(J-t) described in 4.3 also satisfies the conclusion of the lemma. This is because the n+ invariant

406 A. Joseph

elements identify via duality with the n- homology of M(J-L) which is con­centrated on the highest weight vector. This equally applies to the second action coming from right multiplication in U (n -) because of the equivalence ofleft and right multiplication in U(n-) implemented by the principal anti­automorphism. From say [Jl, 2.6] such an action is determined up a choice of polynomial generators for S(n-). Thus we can assume that the second action of n+ (coming from the right action of U(n-» coincides with that constructed above. This is described by the property that for all 0, (3 E R+

, _ {-Na,{3_ae_({3_a) : (3 - 0 E R+ ,

ea(e_{3) - 1 : {3 = 0 ,

o : otherwise

where (ea , e{3-a) = Na,{3-ae{3. We now use (*) to establish the assertion in the last part of 4.3. First

consider the element h of the .6(2) triple (e, h, f) viewed as belonging to the adjoint representation E({3). Since Ie, Ie, h]] = 0 it follows that the image of h under the embedding E({3) '---? 6M({3)~ 6M(O) (isomorphism of U(n+) modules) is a vector a_{3 of weight -{3 satisfying e2 a_{3 = O. (Trivially ea_{3 I- 0 by the analogue of Lemma 1). Let e_a : 0 E R+ denote the choice of polynomial generators for S(n-) satisfying (*) above.

Lemma 2. One has a_{3 = e_{3 up to a non-zero scalar.

Proof. It is enough to show that deg a_{3 = 1. Suppose that deg a_{3 > 1. Applying the e~ : 0 E B gives new elements a_'Y : "Y ~ (3 satisfying e2 a_'Y = o. Assume "Y minimal with the property deg a_'Y > 1. Then deg(e~a_'Y) ~ 1 for all 0 E B. From the remark following Lemma 1 it follows that the leading term of a_'Y is a polynomial in the e_a : 0 E B. Then one may check using (*) that maxaEBdeg(e~a_'Y) = deg a_'Y - 1. This forces deg a_'Y = 2 and "Y to be the sum of two simple roots 0, 0'.

Suppose "Y E R+. View e~ as a differential operator. Then by (*)

Whilst for the first action we must have up to a non-zero scalar that

by the requirement that ea and e~, commute. Then e2 (e_(a+a'» = (ea + ea' )2e_(a+a') = Na,a' + Na',a = o. Yet up to a non-zero scalar one has

Enveloping Algebras: Problems Old and New 407

a_"Y = e_o:e_o:, + ce_(o:+o:,) and so e2 (a_"Y) = 2 which is a contradiction. For 'Y fj. R+ the argument is similar. •

Remark. The conclusion is sensitive to the particular specialization ho: --+

2/(0:,0:) and fails for a different choice.

Corollary. One has e2 (e_"Y) = 0 for all 'Y E R+.

Proof. Apply (*) to the conclusion of Lemma 2. • From this one can easily recover 4.2 (**). Indeed since e is a derivation

of 8(n-) any homogeneous polynomial a_Ii of degree m in e_"Y : 'Y E R+ must satisfy em+1(a_li) = o. To recover 4.2 (**) it is enough to show that em (a_Ii) i= o. Assume (j is minimal with the property that this fails. Applying the e~ : 0: E B it follows that deg e~a_1i < deg a_Ii, 'vi 0: E B. However by the remark following Lemma 1 it follows that a_Ii is a (homogeneous) polynomial of degree m in the e_o: : 0: E B and hence a monomial. Then em(e_li) i= 0 which is a contradiction.

4.5. Now let U, U+, U- be the algebras defined in 3.1. It turns out that there are two mutually commuting actions of U+ on U- in which the invariants reduce to scalars. These obtain from the left and right adjoint actions of U+ on U and then taking the induced action on grFU for an appropriate ad-invariant filtration :F on U which is trivial on U-. All this is explained in some detail in [Jll, , 5.3.1 - 5.3.5]. We claim that one may specialize as below at q = 1 so that U+ becomes U(n+) whilst U­becomes 8(n-) and the above actions become those described in 4.3, 4.4. These specializations were introduced in [JL1, Sect. 6] and reviewed in some detail in [Jll, 3.4.5].

Let A be the localization of the polynomial algebra C[q] at the ideal < q-l >. Let uj denote the A-subring of U+ generated by the Xo: : 0: E B. Then [Jll, 3.4.5 (iii), 4.3.8] one has uj ®A AI < q - 1 > ~ U(n+). Now define a specialization of U- by setting 8A

8A = E9 (8A)-1' I'EP(R)+

with (8A)0 = A and (8A)-1' defined inductively through

Then [Jll, 3.4.5] one may check that 8A is an A subring of U- specializing to 8(n-). Moreover (by construction) it admits a (left) uj action which

408 A. Joseph

on specialization gives an action of U(n+) on 8(n-) in which the elements of n+ become derivations and 8(n-)n+ is reduced to scalars.

Now consider the second action of U+ on U- which commutes with the first and may be viewed as a right action. Since lxo = 0, V a E B it leaves (8A)0 invariant. Then an easy induction argument on weights shows that it leaves 8A invariant. As in [Jll, 3.4.5] one may check that this gives on specialization a second action of U(n+) on 8(n-) commuting with the first so that the elements of n+ become derivations and 8(n-)n+ reduces to scalars. This second action may be identified with that defined by 4.4 (* ).

Set x = EOEBxo.

Proposition. For each 'Y E R+ there exists a unique up to scalars weight vector Y-'Y E U~'Y satisfying x 2y_'Y = o.

Proof. First let f3 be the unique highest root and consider the embedding of the adjoint representation E(f3) <.......t 8M(f3). Here of course the objects refer to Uq(g) which admits [Jll, 4.3] analogous modules. As in the en­veloping algebra case one may identify 8M(f3) with U- as a U+ module (see (Jll, 5.3.3 - 5.3.10]). Then the zero weight space of E(f3) identifies with U"!:f3 and has dimension rk g. On the other hand x2 E(f3)o lies in the direct sum

and it is well-known via Kostant again! that this has dimension rk g - 1. This establishes the existence of a weight vector Y-f3 E U"!:f3 satisfying x 2Y_f3 = O. By 4.4, Lemma 2 it follows that Y-f3 specializes to e_f3.

Now through the right action of U+ on U- one obtains further elements Y-'Y : 'Y ~ f3 in U- satisfying x 2 y_'Y = O. Now for any monomial Xlj in the Xo : a E B the value of (y-f3)Xlj in 8(n-) obtained by specialization is given by the corresponding monomial with reverse order of factors in the e~ applied to e_f3. Then 4.4 (*) gives existence for the remaining roots.

Finally no other weight vector y-'Y : 'Y =1= 0 except multiples of those above can satisfy x2y_'Y = 0 since such a result would also hold in special­ization. I

It is clear that ordered monomials in the y-'Y : 'Y E R+ satisfying the conclusion of the proposition, specialize to the ordered monomials in the e_'Y : 'Y E R+ and hence they are C(q) linearly independent. Comparison of characters [Jll, 3.4.7] shows that they form a C(q) basis for U-. It is however different to the PBW bases described in [L2].

Enveloping Algebras: Problems Old and New 409

It is unfortunately false that the q-character of U- (defined with re­spect to x) coincides with that of S(n-). This may be anticipated from the fact that x is not even a skew-derivation. Thus an ordered monomial in the Y-'Y : 'Y E R+ of degree m is not obviously annihilated by xm+1. In particular in type A2 taking B = {a,,8} one may check that there is no element of weight -(3a + 3,8) in U- annihilated by x4. We remark that x-(a+i3) defined above is just the commutator [Y-a, Y-i3). However already in type A3 with B = {a,,8, 'Y} one has the rather unpleasant expression

for Y-(a+i3+'Y). Of course in specialization the first term becomes e_(a+i3+'Y) whilst the second and third terms cancel.

One may define a quantized enveloping algebra Uq(gc) associated to a Kac-Moody Lie algebra gc and more generally (cf. [U), Chap. 3) for any symmetric matrix C with integer entries. As in the case when gc is semisimple, Uq(gc) is generated over an appropriate torus by positive Xa

and negative Y-a root vectors with a belonging to a "simple" system B of roots. The subalgebra Uq(ne), or simply U-, generated by the Y-a : a E B specializes to the enveloping algebra of a Lie algebra ne and admits an action of the subalgebra Uq(n(j) generated by the Xa : a E B even for non-symmetric C. Set x = EaEB Xa and define m- as the direct sum of its weight subspaces m:,. : J-l E NB given by

As before {a Em:,. I xa = O} reduces to scalars. One may ask if m- pro­vides a PBW basis for Uq(ne). A slightly more natural equivalent formu­lation of this question is whether under the specialization of U- described in 4.5 that m- identifies with the dual of n(j.

It is known that U- is a noetherian domain and its centre has been determined by P. Caldero [Cal as an easy application of the theory devel­oped in [JL2] - see also [JU, Sect. 7] for further details. It appears to admit rather few (even skew) derivations or automorphisms. In type An Alev and Dumas [AD] have determined Fract U- which has a surprisingly simple structure.

410 A. Joseph

References

[A] J. Adams and coeditors, Representation Theory of Groups and Al­gebras, Comtemp. Math. 145 AMS, Providence 1993.

[AD] J. Alev and F. Dumas, Sur le corps des fmctions de certaines algebres quantiques, J. Algebra. (In the Press).

[AK] L. Auslander and B. Kostant, Polarization and unitary representa­tions of solvable Lie groups, Inv. Math. 14 (1971), 255-354.

[Ba] D. Barbasch, Representations with Maximal Primitive Ideal, pp.317-331 in Progress in Math. 92 Birkhauser, Boston 1990.

[BB] A. Beilinson and J. Bernstein, Localization de 9 modules, C.R. Acad. Sci. Paris (A) 292 (1981), 15-18.

[BV] D. Barbasch and D.A. Vogan, Unipotent representations of complex simple Lie groups, Ann. Math. 121 (1985),41-110.

[B1] E. Benlolo, Etude des varietes orbitales dans U(s(n, C», C.R. Acad. Sci. Paris (A), 315 (1992), 537-540.

[B2] E. Benlolo, Sur la quantification de certaines varietes orbitales, Bull. Sci. Math. 3 (1994), 225-241.

[BG] LN. Bernstein and S.1. Gelfand, Tensor products of finite and infi­nite dimensional representatives of semisimple Lie algebms, Com­positio Math. 41 (1980), 245-285.

[Bro] B. Broer, Line Bundles on the Cotangent Bundle of the Flag Vari­ety, Invent. Math. 113 (1993), 1-20.

[Br] R.K. Brylinski, Limits of weight spaces, Lusztig's q-analogs, and fiberings of adjoint orbits, J. Amer. Math. Soc. 2 (1989), 517-533.

[BK] J.-L. Brylinski and M. Kashiwara, Kazhdan-Lusztig conjecture and holonomic systems, Invent. Math. 64 (1981), 387-410.

[BV) D. Barbasch and D.A. Vogan, Unipotent representatives of complex semisimple Lie groups, Ann. Math. 121 (1985),41-110.

[Cal P, Caldero, Sur Ie centre de Uq(n+), Beitrage zur Algebra und Ge­ometrie 35 (1994), 13-23.

fCC] L.G. Casian and D.H. Collingwood, The Kazhdan-Lusztig conjec­ture for genemlized Verma modules, Math. Z. 195 (1987), 581-600.

[C) N. Conze, Algebres d'opemteurs dif]erentiels et quotients des algebres enveloppantes, Bull. Soc. Math. France 102 (1974),379-415.

[CD) N. Conze-Berline and M. Dufio, Sur les representations induites des groupes semisimples complexes, Compositio Math. 34 (1977), 307-336.

[D) J. Dixmier, Enveloping Algebms, 1977.

North-Holland, Amsterdam,

Enveloping Algebms: Problems Old and New 411

[DK] C. De Concini and V.G. Kac, Representations of quantum groups at roots of 1, pp.471-506 in Progress in Math. 92 Birkhauser, Boston 1990.

[DKP] C.De Concini, V.G. Kac and C. Procesi, Quantum coadjoint action, JAMS, 5 (1992), 151-189.

[Dr] V.G. Drinfeld, On almost cocommutative Hopf algebras, Leningrad Math. J. 1 (1990), 321-342.

[Du] M. DuRo, Sur la classification des ideaux primitifs dans l'algebre enveloppante d'une algebre de Lie semisimple, Ann. Math. 105 (1977), 107-120.

[G] A.B. Goncharov, Constructions of the Weil representations of cer­tain simple Lie algebras, Funct. Anal. Appl. 16 (1982), 70-71.

[OJ] O. Gabber and A. Joseph, On the Bernstein-Gelfand-Gelfand reso­lution and the Dufto sum formula, Compositio Math. 43 (1981), 107-131.

IH] W.H. Hesselink, Characters of the Nullcone, Math. Ann. 252 (1980), 179-182.

[HJ] M. Harris and H.P. Jakobsen, Singular Holomorphic Representa­tions and Singular Modular Forms, Math. Ann. 259 (1982), 227-244.

[Ho] R. Hotta, On Joseph's construction of Weyl group representations, Tohoku Math. J. 36 (1984),49-74.

[I] R.S. Irving, The Socle Filtration of a Verma Module, Ann. Ec. Norm. Sup. 21 (1988), 47-65.

(.Ja] J.-C. Jantzen, Moduln mit einem hochsten Gewicht, LN 750, Springer-Verlag, Berlin 1979.

[.H] A. Joseph, A generalization of the Gelfand-Kirillov conjecture, Amer. J. Math. 99 (1977), 1151-1165.

[.12] A. Joseph, Dixmier's problem for Verma and principal series sub­modules, J. London Math. Soc. 20 (1979), 193-204.

(.J3] A. Joseph, Goldie rank in the enveloping algebra of a semisimple Lie algebra III, J. Algebra 73 (1981), 295-326.

(J4] A. Joseph, Completion functors in the 0 category, LN 1020, Springer-Verlag, Berlin 1983, 80-106.

(.T5] A. Joseph, On the variety of a highest weight module, J. Algebra 88 (1984), 238-278.

(.T6] A. Joseph, A surjectivity theorem for rigid highest weight modules, Invent. Math. 92 (1988),567-596.

(.T7] A. Joseph, Multiplicity of the adjoint representations in simple quo­tients of the enveloping algebra of a simple Lie algebra, TAMS 316 (1989), 447-491.

412 A. Joseph

(J8) A. Joseph, On the characteristic polynomials for orbital varieties, Ann. Ec. Norm. Sup. 22 (1989), 569-603.

(J9) A. Joseph, A sum rule and multiplicities for primitive quotients, preprint 1991, unpublished.

(JlO) A. Joseph, Annihilators and associated varieties of unitary highest weight modules, Ann. Ec. Norm. Sup. 25 (1992), 1-45.

(Jll) A. Joseph, Quantum Groups and their Primitive Ideals, Springer­Verlag. (In the Press).

(J12) A. Joseph, Characters for Unipotent Representations, J. Algebra 130 (1990), 273-295.

(JL1) A. Joseph and G. Letzter, Local finiteness for the adjoint action for quantized enveloping algebras, J. Algebra 153 (1992), 289-318.

(JL2) A. Joseph and G. Letzter, Separation of variables for quantized en­veloping algebras, Amer. J. Math., 116 (1994), 127-177.

(JL3) A. Joseph and G. Letzter, Verma module annihilators and quantized enveloping algebras, Ann. Ec. Norm. Sup. (In the Press).

(JL4) A. Joseph and G. Letzter, Rosso's form and quantized enveloping algebras, Preprint 1993.

[Ki) A.A. Kirillov, Unitary representations of nilpotent Lie groups, Us­pehi Mat. Nauk 17 (1962), 57-110.

[K1) B. Kostant, Lie group representations on polynomial rings, Amer. J. Math. 85 (1963), 327-404.

[K2) B. Kostant, On the existence and irreducibility of certain series of representations, Bull. Amer. Math. Soc. 75 (1969),627-642.

[K3) B. Kostant, Quantization and Unitary Representations, pp.87-208 in Lecture Notes in Mathematics 170, Springer-Verlag, Berlin 1970.

[K4) B. Kostant, Verma modules and the existence of quasi-invariant differential operators, pp. 101-128 in Non-commutative Harmonic Analysis, Lecture Notes in Mathematics 466, Springer-Verlag, Berlin 1975.

[LS) T. Levasseur and J.T. Stafford, Rings of Differential Operators on Classical Rings of Invariants, Mem. Amer. Math. Soc. 412 (1989).

[LSS) T. Levasseur, S.P. Smith and J.T. Stafford, The minimal nilpotent orbit, the Joseph ideal, and differential operators, J. Algebra, 116 (1988), 480-50l.

[L1) G. Lusztig, Characters of reductive groups over a finite field, Ann. of Math. Studies, 107, Princeton, 1984.

[L2) G. Lusztig, Introduction to Quantum Groups, Progress in Mathe­matics 110, Birkhauser, Boston 1993.

Enveloping Algebras: Problems Old and New 413

[Ml] A. Melnikov, Robinson-Schensted procedure and combinatorial prop­erties of geometric order s((n), C.R. Acad. Sci. 315 (1992), 709-714.

[M2] A. Melnikov, Irreducibility of the associated variety of simple highest weight modules in s((n), C.R. Acad. Sci. 316 (1993), 53-57.

[M3] A. Melnikov, Orbital varieties and Order Relations on Young Tableaux, Preprint 1994.

[PRV] K.R. Parthasarathy, R. Ranga Rao and V.S. Varadarajan, Repre­sentations of complex semisimple Lie groups and Lie algebras, Ann. Math. 85 (1967), 383-429.

[R] W. Rossmann, Nilpotent Orbital Integrals in a Real Semisimple Lie algebra and Representations of Weyl Groups, pp. 263-287 in Progress in Mathematics 92, Birkhauser, Boston, 1990.

[Ro] M. Rosso, Analogues de la forme de Killing et du theoreme d'Harish-Chandra pour les groupes quantiques, Ann. Ec. Norm. Sup. 23 (1990), 445-467.

[S] N.N. Shapovalev, On a bilinear form on the universal enveloping algebra of a complex semisimple Lie algebra, Funct. Anal. Appl., 6 (1972), 307-312.

[So] J.-M. Souriau, Quantification Geometrique, Commun. Math. Phys. 1 (1966), 374-378.

[V] D.A. Vogan, Unitary Representations of Reductive Lie Groups, Ann. of Math. Studies 118, Princeton 1987.

[T] T. Tanisaki, Characteristic varieties of highest weight modules and primitive quotients, Adv. Studies in Pure Math. 14 (1988), 1-30.

[Z] A. Zahid, Orbite Nilpotent Minimale en type G2 , et Operateurs Differentiels, These, Partie 2, Paris 1991.

The Donald Frey Professorial Chair, Department of Theoretical Mathematics, The Weizmann Institute of Science, Rehovot 76100, Israel and Laboratoire de Mathematiques Fondamentales, Universite de Pierre et Marie Curie, 4 Place Jussieu, Paris 75252 Cedex 05, France

Received April 22, 1994

Integrable Highest Weight Modules Over Affine Superalgebras and Number Theory

Victor G. Kach and Minoru Wakimoto2

To Bertmm Kostant from whom we learned so much

O. Introduction

The problem of representing an integer as a sum of squares of integers has had a long history. One of the first after antiquity was A. Girard who in 1632 conjectured that an odd prime p can be represented as a sum of two squares iff p == 1 mod 4, and P. Fermat in 1641 gave an "irrefutable proof" of this conjecture. The subsequent work on this problem culminated in papers by A.M. Legendre (1798) and C.F. Gauss (1801) who found explicit formulas for the number of representations of an integer as a sum of two squares. C.G. Ba­chet in 1621 conjectured that any positive integer can be represented as a sum of four squares of integers, and it took efforts of many mathematicians for about 150 years before J.-L. Lagrange gave a proof of this conjecture in 1770.

A radically new approach to the problem, based on the theory of elliptic functions, was found by C.G.J. Jacobi in his famous Fundamenta nova (JI published in 1829. In order to state his results, we introduce the following power series in q:

O(q) = Lqi2. jEZ

Then, clearly, the coefficient of qk in the power series expansion of O(q)d is the number of representations of k as a sum of d squares of integers (taking into account the order of summands). Jacobi discovered the following remarkable identities, which gave a solution to the problem for d = 2, 4, 6 and 8:

00

O(q)2 = 1 + 4 L (_1)kqi(2k-l),

j,k=l 00

O(q)4 = 1 + 8 L kqjk, j,k=l

4jk

* Supported in part by NSF grant DMS-9103792.

(0.1)

(0.2)

416 V. G. K ac and M. Wakimoto

00

O(q)6 = 1- 4 E (_I)!U-l)(k+l)(j2 - k2)q!ik, (0.3) j,k=l

j>k,j-k odd 00

O(q)8 = 1 + 16 E (_I)(j+l)kk3qik. j,k=l

(0.4)

Further development has been based, starting with the works of Jacobi and Legendre, on the theory of modular forms. Explicit formulas for odd d ~ 7 have been found by Legendre, Eisenstein, Smith, Hardy and Ramanujan (the case of odd d is much harder than that of even d). In the 1860s J. Liouville obtained a series of beautiful formulas for d = 10 and 12. In 1917 L.J. Mordell showed how in principle the problem can be solved for even d using the Fricke-Klein theory, but the difficulties increase rapidly with d. One can find a detailed (but quite disordered) account on sums of squares in Volume II of Dickson's History of the Theory of Numbers (Chapters VI-IX).

The related problem of representing a positive integer by sums of triangular numbers (i.e., numbers of the form ~j(j + 1), j = 0,1, ... ) has also been quite popular, and a number of deep results was obtained by Gauss, Legendre and others (see Chapter 1 of Vol. 2 of Dickson's "History"). Here the problem, clearly, is to find a power series expansion of powers of the series

00

A(q) = E q!j(j+l). j=O

The main objective of the present paper is to demonstrate that the repre­sentation theory of affine superalgebras is a natural framework for the study of the problem of sums of squares and sums of triangular numbers. However, in order to state our main result we need only the following notions. Let V be a finite-dimensional vector space over lR with a non-degenerate symmetric bilinear form (.1.). Let A be a finite set of vectors in V not containing 0, called the set of roots, and let A = Ao U Al be represented as a disjoint union of two subsets, called the sets of even and odd roots. A choice of a linear function f on V which does not vanish on A defines a subset of positive roots A+ = {a E Alf(a) > O}; we let A,+ = A, n A+, € = 0,1. Given a E V we define the exponential function eU by eU(v) = e(ulv). We call the Weyl denominator associated to the above root data the function

Affine Supemlgebras and Number Theory 417

Let q be an indeterminate; we call the affine denominator the infinite product

where l is the dimension of the lR-span of.6.. Our main objective is to find expansions for Rand R. Of course, in the case when .6. = .6.0 is the set of roots of a simple finite-dimensional Lie algebra, an expansion for R (resp. R) is given by the well-known Weyl denominator identity (resp. Macdonald identity [Mac]), see e.g., [K9]. Our main Theorems 2.1 and 4.1 extend these results to the case when .6. is a set of roots of a finite-dimensional simple Lie superalgebra JJ with a non-degenerate Killing form.

The connection to sums of squares is obvious. Indeed

and if it happens that

l + 1.6.0 1 = 1.6. 11 =: d, (0.5)

then we get an expansion of D(q)d, in view of Gauss' identity

00 1 _ qn

D(q) = I1-. n=l 1 + qn

Evaluating at points other than 0, one gets expansions for triangular numbers (and, more generally, m-gonal numbers).

To be more specific, let us give an example of a root system of type A( m -1, n - 1), where m > n ~ 1. Let V be the (m + n)-dimensional vector space over lR with basis fl, ..• , fm, 01, ... , On and the symmetric bilinear form given by (fi!fi) = -(oiloi) = 1 for all i and j and zero for any other pair. Let .6.0 = {fi - fill ~ i, j ~ m, i # j} U {Oi - Ojl1 ~ i, j ~ n, i # j}, .6.1 = {fi -0i, OJ - fill ~ i ~ m, 1 ~ j ~ n}, .6. = .6.0 U .6.1 . Let .6.+ = {fi - fi, OJ - ojli < j} U {fi - 0ili ~ j} U {Oi - fjli < j}. Condition (0.5) holds iff m = n + 1. In this case, i.e., .6. of type A(n, n - 1), the denominator identity is

(0.6)

418 V. G. K ac and M. Wakimoto

Here Xi = e- f ', Yi = e- li" i and j take all possible values and €( w) is the sign of the permutation w E Sn+1' Furthermore, the affine denominator identity reads:

(0.7) Here Mn is the sublattice of the lattice 'Ei /l€i of vectors with zero sum of coordinates (= the root lattice of type An) and xm := X;"1 ... x;:'.;t. Dividing both sides of (0.7) by Rn and taking the limit as Xi, Yi --+ 1, we obtain an expansion of O(q)2n(n+1). One of the ways to actually compute this limit is explained in the proof of Theorem 4.2, but probably there is a different way giving a better formula. Of course, for n = 1 we thus recover Jacobi's formula (0.2). Another specialization of (0.7) gives the following expansion (see Example 4.3b):

11.+1 "",11.+1 2 _ n ( _ )-1 O(_q)2n = 2n I: I:(-l)ls lq-r 6'=1 (m i +mi -mi sd I1 1 + q(n+1)(m j -s j )

mEMnsES j=1

(0.8) where S is the set of all subsets of the set {€i - €j Ii < j} (= set of positive roots of type An), and s (resp. lsI) denote the sum (resp. number) elements from s E S. Formula (0.8) for n = 1 is an equivalent form of (0.1).

Even the simplest case of (0.6), when n = 1, is a truly remarkable identity. Letting u = -xI!YI, V = -x2/YI, it takes the following beautiful symmetric form:

(0.9)

After establishing this identity, we realized that it has been rediscovered many times in the past. It is implicitly contained in [KL] as it follows immediately by comparing characters in the super boson-fermion correspondence (which gives a combinatorial proof of (0.9», and in [M] where it is used to rewrite the characters of N = 2 superconformal algebras in a multiplicative form. As has been pointed out in [F] and [H], (0.9) is a special case of the Ramanujan summation formula for the bilateral basic hypergeometric function I WI.

Condition (0.5) holds iff 11 is of type A(n, n -1), B(n, n) or D(n + 1, n + 1) (n = 1,2, ... ), which gives an expansion for O(q)N for N = 2n(n+l), 2n(2n+l)

Affine Supemlgebms and Number Theory 419

5.4). The first members of the first and the second series are Jacobi's formulas (0.2) and (0.3). The second member of the first series gives an expansion for D(q)12, however we do not know how to derive from this expansion the famous result of J. Liouville (1860) that the number of representations of 2k, where k is odd, as a sum of 12 squares equals 264 Ej1k jS. Finally note that Jacobi's formula (0.4) is obtained by evaluating at 0 the (conjectured) denominator identity for the case 9 = gl(2, 2), which is a Lie superalgebra with a zero Killing form (see Conjecture 7.1). Another very interesting case which is not covered by Theorem 4.1 is 9 = Q(m) (see Conjecture 7.2). A specialization of the corresponding affine denominators gives very beautiful power series expansions of ~(q)482 and ~(q)48(8+1) (see Section 7). For s = 1 we thus recover Legendre's formulas for sums of 4 and 8 triangular numbers (whereas formulas for 2, 4 and 6 triangular numbers are obtained by certain specializations of the affine denominator identities for A(l, 0), A(l,O) and B(l, 1) respectively, see Section 5). For s ;::: 2 we obtain formulas which are probably new. For example, we find that the number of representations of n as a sum of 16 triangular numbers equals

1 3.43

L ab(a2 - b2)2. a,b,r.,sEN odd

a>b ar+bs=2n+4

In Section 1 we recall the necessary material on a simple finite-dimensional Lie superalgebra 9 [K2J. The new material here concerns the dual Coxeter number h v (which is the eigenvalue of the Casimir operator associated to the suitably normalized Killing form). As in the Lie algebra case, h v plays an im­portant role in the theory of affine superalgebras. In Section 2 we introduce the important notion of the defect of 9 (which is 0 in the Lie algebra case) and of a maximal isotropic set of roots. This allows us to state the denominator identity (Theorem 2.1) for the root system of g. We give a proof which works in the case of def 9 ::5 1 (which takes care of all exceptional Lie superalgebras). A proof in the general case involves more calculations and will be published elsewhere. At any rate, the proof is based on the analysis of irreducible subquotients of the Verma module M(O) over 9 with highest weight O. Since the denominator iden­tity may be viewed as the character of the I-dimensional g-module, it is only natural to go on and try to compute in Section 3 the character of an arbitrary finite-dimensional irreducible g-module V. There has been a number of papers concerned with this problem [K3, K4, BL, TMl, TM2, VJl, VJ2, JHKT1, JHKT2, P, PSI, PS2, S, Se, ... J. Our basic new ingredient is the notion of a A-maximal isotropic subset, which allows us to define atypicality of V which is independent of the choice of the set of positive roots, and of a tame g-module V. Our first basic result is Theorem 3.1 which states that V is tame (and hence gives a formula for ch V) in the case when for some choice of the set of positive

420 V.G. Kac and M. Wakimoto

roots there exists a A + p-maximal isotropic subset consisting of simple roots. (Note that Theorem 2.1 is a special case of Theorem 3.1 since def 9 = atpO.) This theorem covers most of the known formulas for ch V (proved in [K3, K4, BL, VJ1, VJ2, JHKT2, ... j), but unfortunately there exist V that are not tame. We describe all cases of non-tame V of atypicality ::s 1; this gives, in particular, character formulas for all V over exceptional Lie superalgebras (Ex­ample 3.3). We also give a proof of a dimension formula (Theorem 3.2), which is based on a nice regularization procedure, and of a superdimension formula (Theorem 3.3), which is based on an interesting property of the root system of 9 (Lemma 2.2d) and on Theorem 2.1.

In the second part of the paper (Sections 4-8) we turn to the associated to 9 affine superalgebra g. The main result is Theorem 4.1 which gives the affine denominator identity in the case when the Killing form is nondegenerate (equivalently, hV =I- 0). The proof is again based on the analysis of irreducible subquotients of the Verma module over 9 with zero highest weight. (Of course, this is the basic idea of the proof of Macdonald identities given in [K1].) Ex­amples 4.4 and 4.5 give a more convenient form for specializations of the affine denominator identity. Theorems 4.2 and 4.3 and Example 4.3b give explicit formulas for two specializations of this identity. These formulas are applied in Section 5 to derive expressions for the number of representations of an integer as sums of squares and sums of triangular numbers. In Section 6 we use the simplest affine denominator identity to study transformation properties of cer­tain "degenerate" theta functions and modular forms and apply this to prove transformation properties of the N = 2 superconformal algebras, c/. [RY). (These results may be extended to the general case by the same method.) In Section 7 we discuss the affine denominator identity for 9 = gl(2, 2), the sim­plest case when hV = o. Another extremely interesting case is 9 = Q(m). We conjecture the corresponding affine denominator identity and derive formulas for the number of representations as sums of triangular numbers in terms of dimensions of irreducible representations of Slm+l. Finally, in Section 8 we introduce the notion of an integrable g-module, based on our experience with the I-dimensional module. Theorem 8.1 gives a classification of irreducible integrable highest weight g-modules L(A).

(As in the Lie algebra case, the level of such a module is a non-zero integer, but there may be infinitely many such modules of positive level.) It is an extremely interesting problem to calculate the characters of integrable L(A). A partial answer is given for A(l,O)" (Example 8.1). Of course, in the case when def 9 = 0 the answer is well known for both 9 and 9 [K5].

Acknowledgment. We learned from V. Serganova the very useful method of odd reflections. Discussions with her and I. Penkov were very illuminating. We relied extensively on the (unpublished) tables of J. Thierry-Mieg [TM1] and computer calculations of J . Van der Jeugt of dimensions and superdimensions

Affine Supemlgebras and Number Theory 421

computer calculations of J. Van der Jeugt of dimensions and superdimensions of irreducible 9-modules. Discussions with them and R.C. King were very useful. We are very grateful to all these people.

The results of this work were reported at the conference on universal en­veloping algebras in Paris in June 1992, and at Kostant's conference at MIT in May, 1993.

1. Dual Coxeter number

Let 9 = 90 + 91 be a finite-dimensional simple Lie superalgebra over C with a non-degenerate even invariant bilinear form B. Recall that "even" (resp. "in­variant") means that B(gO,gl) = 0 (resp. B([a, bl, c) = B(a,[b,c]),a,b,c E g) and that all such forms are proportional. Denote by h'b the eigenvalue of the Casimir operator OB associated to B [K2, p. 851 in the adjoint representation.

Proposition 1.1. h'tJ =1= 0 if and only if the Killing form is non-degenerate.

Choose a Cartan sub algebra I) of go, let D. E 1)* be the set of roots, D.o and D.l the subsets of even and odd roots and let W c GL(I)*) be the Weyl group (= the group generated by reflections ret with respect to even roots a).

Let g = I) E9 ( E9 get) be the root space decomposition. aEA

Recall that an odd root a is isotropic (Le. B(a, a) = 0) iff 20' is not a root. We let

~o = {a E D.o I ~a i D.}, ~l = {a E D.l I (ala) = o}.

Then D.l \ ~ 1 = ~ (D.o \ ~o) and ~o is a root system. Choose a set of positive roots D.+ in D., let D.£+ = D.£ n D.+ (f = 0 or 1)

and let n+ = E9 get . Let n = {at, ... , at} C D.+ be the set of simple roots and let 0 E D.,;15kthe highest root. Let P£ be the half of the sum of the roots from D.£+ and and let P = Po - Pl. Then, B(p,ai) = ~B(ai' ai), i = 1, ... , l. Furthermore one has

h'tJ = B(p,p) + ~B(O,O). (1.1)

The following generalization of Freudenthal-de Vries "strange formula" holds:

(1.2)

Assuming that h'tJ =1= 0, let

D.~ = {a E D.o I h'tJB(a,a) > o}

422 V.G. Kac and M. Wakimoto

invariant bilinear form on 9 normalized by the condition

(ala) = 2 for the longest root a E L).~. (1.3)

The corresponding to this form number h v = hG1.) is called the dual Coxeter number of the Lie superalgebra g. We let h v = 0 if h'j, = o. We give below the number hV in all cases (excluding Lie algebra cases):

A(m,n): hV=lm-nlj C(n):hv =n-1j

B(m,n): hV=2(m-n)-lifm>n, =n-m+~ifm:::;nj

D(m,n): hV =2(m-n-1)ifm;;::n+1, =n-m+1ifm<n+1j

D(2,lja): hV=Oj F(4):hv =3j G(3):hv=2.

Remarks 1.1. (a) Let 1\:( . , .) denote the Killing form, which we assume to be non-degenerate. Then L).~ = {a E L).o 11\:( a, a) > O} is the root system of one of the simple components of go and hV = I\:(a, a)-1 for the longest a E L).~.

(b) hV = 0 if and only if 9 is of type A(n, n), D(n + 1, n) or D(2, 1j a) [ j. In these cases we define L).~ as follows. The root system L).o is a union of two orthogonal to each other root subsystemj namely one has respectively:

Then we let L).~ to be the first subset and W~ to be the subgroup of W generated by the rex with a E L).~. We normalize (.1.) by (1.3) as before.

(c) There are many W-inequivalent choices of L).+. However, there is a unique (up to an automorphism of the root lattice) choice of L).+ such that II contains a unique odd rootj we call this L).+ standard.

In order to prove the above statements, recall that [K2, p. 85j the eigenvalue of the Casimir operator nB in a representation with highest weight A is equal to B(A, A + 2p) and that when the Killing form I\: is non-degenerate one has:

I\:«(}, (} + 2p) = 1 (1.4)

provided that dim 90 =1= dim 91. Since

dim go = dimgl iff 9~A(n+1,n),B(n,n)orD(n,n) (1.5)

and since (1.4) is a polynomial identity which holds for all other values of (m, n) (except when I\: == 0) we obtain that (1.4) holds in all cases when", is non-degenerate. This proves Proposition 1.1 and formula (1.1).

From formula (2.1) in Section 2, it follows that

B(p, p) is independent of the choice of L).+. (1.6)

Affine Superalgebras and Number Theory 423

Hence it suffices to check (1.2) for one choice of ~+. A more conceptual proof of (1.2) is based on the theory of "degenerate" theta functions (cf. Remark 6.1 in Section 6).

Example 1.1. g = A(m,n) with m > n. Choose ~+ such that the number of odd simple roots is maximal. Then II = II' U II", where (ala) = 0 if a E II' and (ala) = 2 if a E II", and one can order the sets II' and II":

in such a way that all non-zero scalar products of distinct roots are:

(a2;-lla2;) = 1 if 1 ~ j ~ n + 1, (a2;la2j+d = -1 if 1 ~ j ~ n,

(a;laj+l) = -1 if 2n + 2 ~ j ~ m + n.

In particular, II' (resp. II") is a set of simple roots of A(n+l, n) (resp. Am-n-d. The root system ~~ is of type Am and has the following set of simple roots: {at + a2, ... ,a2n+t + a2n+2,a2n+3, ... ,am+n+d. In the (most important for us) case when m = n + 1, we have II = II' and ~~ has simple roots {al + a2, ... , a2n+1 + a2n+2}. Note that (1.2) for A(m, n) follows from the "strange formula" for Am - n - 1 •

2. The Weyl denominator of g

Given a subset of positive roots ~+ of ~ and a simple root a, we may construct a new subset of positive roots (cf. [PS2j):

~~ = ra(~+) if (ala) i= 0,

~~ = (~+ U {-a}) \ {a} if (ala) = o.

The set ~~ is called a simple reflection of ~+. Then one has:

, = {p - a if (ala) i= 0 p p+a if (ala) = o. (2.1)

Note that ~~ and ~+ are W-inequivalent if (ala) = o. By a usual argument, any two subsets of positive roots of ~ may be obtained from each other by a sequence of simple reflections.

By a regular exponential function on ~ we mean a finite linear combination of exponentials e\ >. E ~*. A rational exponential function is a ratio P / Q where P and Q are regular exponential functions and Q i= o. The Weyl group W acts on the field of rational exponential functions via e A 1--+ eW(A) , W E W.

424 V.C. Kac and M. Wakimoto

The rational exponential function

(2.2)

is called the Weyl denominator of g. Introduce also the Weyl superdenomina­tor:

(2.3)

The proof of the following lemma is straightforward.

Lemma 2.1. Let A+ and A~ be two subsets of positive roots in A, and let R andR' (resp. R andR') be the corresponding Weyl (resp. super) denominators. Then

The basic difference between the Lie algebra and superalgebra cases is that the restriction of the bilinear form (.\.) to the subspace V:= E IRa may be

aE.o. indefinite; the dimension of a maximal isotropic subspace of V is called the defect of 9 and is denoted by def g. Here is a list of defects

A(m -1, n -1), B(m, n), D(m, n)

C(n), D(2, 1; a), F(4), G(3)

def 9 = min(m, n)

def 9 = 1.

(Note that def 9 = D iff 9 is a Lie algebra or a Lie superalgebra B(D, n).) A subset S of A is called maximal isotropic if it consists of def 9 roots that

span a maximal isotropic subspace of V. The following can be checked by a straightforward calculation.

Lemma 2.2. (a) A maximal isotropic set of roots always exists.

(b) Given a maximal isotropic subset of roots S there exists A+ such that SeTI.

(c) Given two maximal isotropic subsets of roots S = {PI,"" Ps} and S' =

{,B~, ... , P~}, there exists w E WU such that w(,BD = ±,BrT(i) for a permutation a.

Affine Supemlgebms and Number Theory 425

(d) Let S C ~l+ be a maximal isotropic subset and let ~g+ = {o: E ~0+1 o:..LS}. Then

Theorem 2.1. Let S be a maximal isotropic subset of roots and choose ~+ such that S C TI. Then

(2.4)

This theorem is a special case of Theorem 3.1 in Section 3. Here we shall give a proof in the case def g = 1. The proof is based on the

following

Lemma 2.3 Suppose that an irreducible g-module with highest weight A E ~*

is a subquotient of a Verma g-module M(A). Then there exists a sequence of weights AO = A, Al, ... , Ak = A of M(A) such that Ai (0 < i :S k) is obtained from Ai-l in one of the following two ways:

(i) Ai = Ai_l - WYi, where Ii E ~o+ and (Ai-l + pi,i') = n E N, (ii) Ai = Ai-l - Ii, where Ii E ~l+ and (Ai-l + pi,i) = O.

Proof. See [K4, Theorem 3b]. (Note that the exponents in the determinant formula given by [K4, Theorem 3a] should be corrected as in [KB].) •

Proof of Theorem 2.1 (case def g = 1.) As usual, we have:

ePR = L coX eoX+p , CoX E Z, CoX i= 0, (2.5) oXEF

where F is the set of weights described by (i) and (ii) in Lemma 2.3 for A = o. Choose ~+ such that TI contains a unique odd (isotropic) root, say, f3. We clearly have:

(2.6)

Consider the minimal i for which the possibility (ii) of Lemma 2.3 might occur. Then Ai-l = w(p) - p, where w = r'Yi_l ... r,Yl E Wand Ai = Ai-l - Ii, (plw-1(fi)) = 0, Ii E ~l. But by a case-wise inspection, (ph) = 0, I E ~l => I = ±f3. Hence Ii = w(f3). Consider now the minimal j :S i-I for which Ii ¢ ~~, i.e., Ii = O. But (pIOV) < 0, hence we conclude that w E W". Thus, Ai := w(p - f3) - p, w E W".

426 V.G. Kac and M. Wakimoto

FUrthermore, (,8I(JV) 2: 0, hence (p - ,8I(JV) < 0, hence if s is minimal > i for which possibility (ii) of Lemma 2.3 might occur, we conclude that A8-1 = Wl{P - (3) - P, WI E WU.

By a case-wise inspection, we see that (p- f31(31) = 0, f31 E ~1+ '* f31 = f3. Hence A8 = Wl(P - 2(3), etc. Thus, the set A := {A + piA E F} coincides with the set {w{p - j(3)lw E W~, j E Z+}. But by Lemma 2.1, w{ePR) = €{w)ePR. This together with (2.6) completes the proof. •

Remark 2.1. Recall that €( w) = (_l)m, where m is the number of reflections with respect to roots from D.o+ entering in a decomposition of w as a product ofreflections. We let t{w) = {_l)ih, where in denote the number ofreflections occurring in this decomposition with respect to roots from ~o+. Then one easily deduces from (2.4) the formula

(2.7)

3. Finite-dimensional irreducible g-modules

Let A E ~*. The atypicality of A, denoted by atp A, is the maximal number of linearly independent roots f3i such that (f3i lf3j) = 0 and (AIf3i) = 0 for all i and j. Such a set {f3i} is called a A-maximal isotropic subset of D.. (For example, a maximal isotropic subset S c 11 is a p-maximal isotropic subset.)

Proposition 3.1. Given two A-maximal isotropic subsets {f3;} and {f3H of D., there exists w E W such that w(>.) = >. and W(f3i) = ±f3~(i) for some permutation (j.

Let V be a finite-dimensional irreducible g module. If we fix a set of positive roots D.+, we may talk about the highest weight A of V and about the p. If D.~ is obtained from D.+ by a simple o:-reflection, where (0:10:) = 0, and if A' is the highest weight for D.~, then we clearly have

A' = A - 0: if (Alo:) f 0; A' = A if (Alo:) = o.

Hence by (2.1) we obtain:

A' + p' = A + p if (A + plo:) f 0,

A/+P'=A+p+o: if (A + plo:) =0.

(3.1)

(3.2)

Corollary 3.1. For the g-module V, atp(A + p) is independent of the choice of D.+.

Affine Supernlgebms and Number Theory 427

Thus, we may talk about the atypicality of the g-module V: atp V :=

atp(A + p), which is independent of the choice of ~+. Note that atp V ~ def g. Choose a A + p-maximal isotropic subset SA in ~+ and consider the fol­

lowing rational exponential function:

EA:= ~ f(w)w(eA+P II (l+e-f3)-I). wEW f3ESA

EA:= ~ l(w)w(eA+P II (1- e-(3)-I)' wEW f3ESA

Denote by jA the coefficient of eA+p in the regular exponential function TIaEA1+ (1 + e-a ) LA and, provided that jA "I 0, let

h ·-1 -PR-I ", h ·-1 -PRY-Ii' C A = JA e '-'A, sc A = JA e '-'A

dimA = chACO), sdimA = schA(O).

We shall write sometimes chA,Ll.+,sA' ... in place of chA, ... to emphasize the dependence of ~+ and SA.

Definition 3.1. The g-module V is called tame with respect to a set of positive roots ~+ and a subset SA C ~+, where A is the highest weight of V with respect to ~+ and SA is a A + p-maximal isotropic subset of ~+, if jA "I 0 and the character of V is given by

(3.3)

Remarks 3.1. (a) Formula (3.3) implies

Here + or - occurs according as the highest weight vector is even or odd. (b) If V is tame, then dim V = dimA,sdim V = ±sdimA.

Remarks 3.2. (a) Suppose that ~~ is obtained from ~+ by a simple a­

reflection, where (ala) = O. If (A + pia) "10, then chA,Ll.+,sA = chA',Ll.+,sA. If (A + pia) = 0 and a E IInSA, we let S~ = (SA \ {a})U{-a}j then chA,Ll.+,sA = chA,Ll.+,s~. This follows from (3.1) and (3.2). However, if (A + pia) = 0 and a E II \ SA, chAI,Ll.+'SAI may be different from chA,Ll.+,sA.

(b) Let SA = {/h, ... ,.8s} and suppose yEW is such that yeA + p) = A + p, y(.8i) = .8i for i = 2, ... , s, and y(.81) = f(Y).8~, where .8~ E ~+ . Let S~ = {.8~,.82, ... ,.88}. Then ChA,4+,SA = chA,Ll.+,s~.

428 V.G. Kac and M. Wakimoto

(c) Theorem 2.1 states that the I-dimensional g-module is tame with respect to (~+, So) if II :J So. Note that then

jo = IW/W~I· (3.4)

The number jo is listed below: A(m, n)(m ~ n) : (n + 1)!; B(m, n) : n!2n if m > n, 2mm! if m ::; n; C(n) : 1;

D(m, n) : n!2n if m ~ n+ 1; 2m - 1m! if m < n+ 1; D(2, 1; a), F(4), G(3) : 2.

Proposition 3.2. Let V be a finite-dimensional irreducible g-module. Choose a set of positive roots ~+ and let A denote the highest weight of V.

(a) (ef. [K4]) Let V be a typical g-module, i.e., (A + pia) =1= 0 for a E .b.1

(equivalently: atp V = 0). Then V is tame with respect to (~+, 0).

(b) (cf. [BLj, (JHKT2, VJ2j) If g is of type A(m, n) or C(n) and atp V = 1, then V is tame with respect to (~+, SA) for any choice of SA and jA = 1. (Note that atp V = 0 or 1 if g is of type A(m,O) or C(n).)

(c) If (A + pia) > 0 for all a E ~o+ or (A + pia) ~ 0 for all a E ~o+ and atp(A + p) = 1 then jA = 1.

Proof. If V is a typical g-module, then formula (3.3) with SA = 0 and some jA > 0 follows from the description of singular weights of the Verma module M(A) and W-anti-invariance of ePRch V (cf. [K4]). But if jA =1= 1, then the stabilizer of A + p in W is non-trivial, and since it is generated by reflections, we would get that eh V = 0, which is impossible, proving (a).

(c) is clear and (b) follows from [BL, JHKT2, VJ2] (or as above). •

Theorem 3.1. Let V be a finite-dimensional irreducible g-module with highest weight A with respect to a subset of positive roots ~+. If SA C II C ~+, then the g-module V is tame with respect to (~+, SA).

Note that Theorem 2.1 follows from Theorem 3.1 and Remark 3.2b.

Example 3.1. If atp V = 0, then SA = 0 and Theorem 3.1 gives the well­known character formula of a typical g-module (cf. Proposition 3.2a).

Example 3.2. If atp V = 1 and g is of the first kind, i.e., of type A(m, n) or C(n), then one can always choose ~+ such that SA c II, hence by Theorem 3.1 the g-module V is tame (cf. Proposition 3.2b). There exist, however, modules of atypicality 3 over A(4,3) that are not tame for any choice of ~+ and SA (examples were provided by J. Van der Jeugt).

Example 3.3. Let now g be of the second kind, i.e., oftype B(m, n), D(m, n), D(2, l;a), G(3) or F(4). We exclude B(O,n) for which all V are typical. Let ~+ be a standard set of positive roots. Here are the corresponding Dynkin

Affine Superolgebms and Number Theory

diagrams [K2, p. 56]:

B(m,n),m > 0 B(O,n)

o - 0 - ... - 02 - 02 - ... - 02 => 02 0-0-···-0-0=>0

D(m,n),m ~ 2 0 - 0 - ... - 02 - 02 - ... - 02

01 D(2,1;a) 20

G(3) F(4)

01 02 - 04 E:02 02 - 030(:= 02 - 01

429

Let 0 be the only simple root of Cio+ which is not in II; its coefficients in terms of II are given above (note that 0 = () in all cases except for B(m, n) and D(m, n) with n > 1). Let b = -(pIOV) (as usual, a V := 2a/(ala) for a E Cio); the values of b are as follows:

B(m,n): m -!, D(m,n): m -1, D(2, 1: a): 1, G(3): ~, F(4) : 3.

Let V be a finite-dimensional irreducible g-module and let A be its highest weight. Let k = (AIBV). Recall that dim V < 00 implies that k i := (Alan E Z+ for all ai E II n Cio and that k E Z+. For k > b these conditions are sufficient, and for k ::; b there are some extra conditions listed in [K2, p. 84] (note that b there slightly differs from b here). Suppose that atp V = 1 (this is always the case for a non-typical V over 9 of type B(m, 1), B(I, n), D(m,I), D(2, 1; a), G(3), and F(4», and let SA = {a} (the choice of fJ may not be unique). We have: jA = 1 if k ~ band jA = 2 if k < b. If the g-module with the highest weight To(A + p) - p is not finite-dimensional (which is always the case when k ::; b or k > 2b), then V is tame, i.e.,

chV = ChAo

Otherwise we have:

ch V = ChA - Chr,,(A+p)_p .

Furthermore, provided that the highest weight vector is even, we have:

where W E W is the shortest element such that w(fJ) E IT. Here is a complete list of non-tame modules over exceptional Lie superalgebras (we write A =

430 V. G. K ac and M. Wakimoto

(k; k2' k3 , •• • »: D(2, 1;0'): G(3) :

F(4) :

A = () = (2; 0, 0) : ch V = cho -1.

A = (3; 0, n) : ch V = ch(3;O,n) - ch(2;O,n) . A = (5; 0, 0) : ch V = ch(5;o,o) -1. A = (4; 0, n, 0) : ch V = ch(4;O,n,O) - ch(2;O,n,o) . A = (6; 0, 0, 0) : ch V = ch(6;o,o,O) -1.

In the case of D(2, 1;0') this result is consistent with [VJ1j.

A proof of these results will be given elsewhere. Here we note that the same proof as of Theorem 2.1 shows that for g of defect 1 and atypical V with the highest weight A such that (A + pl,8) = 0, ,8 E .3.1+ and such that k < b

for g of the second kind one has:

ePRch V = E E(W)W (eA+P /(1 + e-J3». (3.5) wEWd

Given integers a and b such that -1 :S a :S b, we let

Let Ne = l~e+l, E = ° or 1.

Theorem 3.2. Let A E I:J*, and let SA = {,8I' ... ,,88}' Then (here and further Ikl stands for Ei ki ):

The proof of the theorem is based on the following simple lemma.

Lemma 3.1. Let P( Xl, ..• , X 8) be a polynomial of degree :S d in the indeter­

minates Xl. ••. ,X8 •

(a) In the domain IXil < 1 one has

( ~) 8 xnj

" (-I)lnla;rtP(n) =" "(_I)lkl ~ P(k) IT j.. L..J L..J L..J k (1 + X .)n, +1 nEZ' nEZ' kEZ. j=l J + + +

/nl:fd ki:5ni

(3.7) Here and further we use the usual notation:

Affine Supemlgebras and Number Theory 431

(b) The value of the rational function on the right-hand side of (3.7) at Xl = ... = Xs = 1 equals

2-s - d L (-l)liiIC(litl + s - 1, d + s - l)P(it). (3.8) nEZ+ ini$d

Proof. It suffices to prove (3.7) for a polynomial P(x) = (;), 0 ~ j ~ d.

But this follows from the well-known formulas:

'"' n (n) (-x)j L...J (-x) j = (1 + x)i+l

nEZ+

in the domain Ixl < 1.

(b) follows from (a) and the formula

'"' (it) (N+S-1) L...J k = Ikl + S - 1 . nEZ+ ini=N

•

Remark 3.3. It is clear from the proof that one can take the summation over it in the right-hand side of (3.7) and in (3.8) over the set [P] E Z~ defined as

follows. We let [XU] = {b E Z~lbi ~ ail and let [P] be the union of [XU] over all the monomials occuring in P with a non-zero coefficient.

Proof of Theorem 3.2. Given A E lJ*, define the specialization Ft>. by

First, we compute the following specialization, where IXi I < 1 for all i and

Ro = TIo:E.t.o+ (1 - e-O:):

L £(w)et (w(A+p-n tlh-... - n .f3s)lpo)

= L (-1) Inlxn -=-wE=..:.W-'--_--=--:----:=-:-__ _ Ftpo(ePoRo)

432 V.G. Kac and M. Wakimoto

Hence we have in the domain IXil < 1:

(3.9)

Note that the expression under the limit at the left-hand side of (3.9) is a continuous function in the domain It I < E and IXi -11 < E for sufficiently small E. Hence we may calculate

by applying Lemma 3.1a to the right-hand side of (3.9) and then evaluating it at Xl = ... = Xs = 1 using Lemma 3.2b. Since dimA = limt-->o Ftpo chA, we obtain the result. •

Theorem 3.3. Let A E ~*, let 8 A be a A + p-isotropic subset and let s = 18 A I. Consider the root system MA := {a E L\ola ..1 ambda}, let p~ be the half of the sum of roots from Mj; := MA n ~+ and let LA (A) denote the irreducible module with highest weight A over the reductive Lie algebra with root system MA. Let a = n - m - 1 for 9 of type B(m, n) with m < n and let a = 0 in all other cases. Then

IsdimAI = IW/W~12-ajAI dimLA(A + p - p~) if s = def g, (3. lOa)

sdimA = 0 otherwise. (3.lOb)

Proof. Let WA be the subgroup of W generated by reflections with respect to the roots from MA and let WI be a set of left coset representatives, so that W = WI WA. Due to (2.3), formula (3.4) can be rewritten using the Weyl character formula for the root system MA as follows:

For A E ~* such that (Ala) =f. 0 for all a E ~, we apply to both sides of this

Affine Supemlgebms and Number Theory 433

equality the specialization po. to obtain as t ........ 0:

IT (Ala) sdimA =

aEAo+\A1+

t IA1+I+IMtl-IAo+I-IAI ( E l(w) IT (Alw(a))) dimLA(A+p-p~)+o(t). WEWl aEMt\A

Due to Lemma 2.2d, if s < def 11, the right-hand side tends to 0 as t ........ 0, proving (3.lOb), and if s = defg, we obtain as t ........ 0:

(3.11)

where

IT (Ala) E l(w) IT (Alw(a)) (3.12) wEW1

is a constant which, in fact, is independent of the choice of A (since all other fac­tors in (3.12) are independent of A). It follows that the constant C = C(A, .6.+, A) is independent up to a sign of the choice of (.6.+, A) as well. Recall now that, according to Theorem 2.1, the I-dimensional module is tame with respect to a certain pair (.6.~, S'). Applying (3.11) to the I-dimensional g-module (for which sdim = ±1) and using (3.4), we obtain:

Finally a case by case inspection shows that dim Lo(p' - pg') = 2a , proving (3.lOa).

In the case g = C(n) a formula equivalent to (3. lOb) was obtained in [VJ2] .

• Conjecture 3.1. sdim V = 0 if and only if atp V < def g.

Example 3.4. (a) if V is a typical g-module then formulas (3.6) and (3.10) turn into the known formulas [K4]:

dim V = 2Nl IT (A + pia), aELl.o+ (PolO')

sdim V = 0 if 11 -:f B(O, n), = IT (A + pla)j(pla) if g = B(O, n). aEAo+

(b) Let g be an exceptional Lie superalgebra and let V be a finite-dimensional irreducible non-typical g-module with highest weight A, so that atp V = 1. We keep the notation of Example 3.3.

434 V. G. K ac and M. Wakimoto

9 = D(2, l;a) : (a1lad = 0, (a1Ia2) = -1, (a1Ia3) = -a, (a2Ia2) = 2, (a3Ia3) = 2a, (a2Ia3) = O. Assume that V is neither the I-dimensional nor the adjoint module. Then sdim V = ±2 and ~ dim V is given by the following formulas, where m = k2, n = k3:

4kmn + 2k(m + n) - 2n + 1 if f3 = a2,

4kmn + 2(km + 3kn - mn) + 4(k - n) - 1 if f3 = a1 + a2,

4kmn+ 2(3km+ kn - mn) +4(k - m) -1 if f3 = a2 + a3,

4kmn + 2(3km + 3kn - mn) + 4(2k - m - n) - 7 if f3 = a1 + a2 + a3.

(Recall that for a typical V one has [K4]: dim V = 16(m + 1)(n + 1)(k - 1) and sdim V = 0.)

9 = G(3) (m = k2' n = k3): If f3 = at, then dim V = 1; f3 =1= a1 + a2. If f3 = a1 + a2 + a3, then k = 2, m = 0 and sdim V = 2n + 3. If f3 = a1 + 3a2 + a3, then 2k = m + 6; for k = 3 we have sdim V = -2n - 3, for k > 3 we have sdim V = -2(m + 2n + 3). If f3 = a1 + 3a2 + 2a3, then 2k = m + 3n + 9 and sdim V = 2(m + n + 2). If f3 = a1 + 4a2 + 2a3, then 2k = 2m + 3n + 10; for k = 5 we have dim V = 321, sdim V = -1, for k > 5 we have sdim V = -2(n + 1).

9 = F(4) (I = k2' m = k3, n = k4): If f3 = at, then dim V = 1; f3 =1= a1 + a2. If f3 = a1 + a2 + a3, then k = 2, 1= n = 0 and sdim V = (m + 2)3. If f3 = a1 + 2a2 + a3, then 3k = 1- 2n + 8; if f3 = a1 + a2 + a3 + a4, then 3k = 2n - I + 10; in both cases k ~ 3 and sdim V = -em + 1)(1 + m +n + 3)(1 + 2m + n+ 4). If f3 = a1 + 2a2 + a3 + a4, then 3k = I +2n+ 12; if k = 4, then 1 = n = 0 and sdim V = (m+2)3; if k > 4, then sdim V = (l+m+2)(m+n+2)(1+2m+n+4). If f3 = a1 +2a2+2a3+a4, then 3k = I +4m+ 2n + 16 and sdim V = -(I + m+ 2)(n+ 1)(l + m+ n + 3). If f3 = a1 + 3a2 + 2a3 + a4, then 3k = 31 + 4m + 2n + 18; if k = 6, then I = m = n = 0 and sdim V = 1; if k > 6, then sdim V = (m+ l)(n+ 1)(m+n+2).

4. The denominator identity for affine superalgebras

and its specializations

Let g, as before, be a finite-dimensional simple Lie superalgebra over C of rank f with a non-degenerate even invariant bilinear form (.1.) (see Section 1). Let 9 = qt, r 1] ® 9 ffi CC ffi Cd be the associated to 9 affine super algebra. The Z2-gradation of 9 extends from that of 9 by letting deg t = 0, deg C = deg d = O. Denoting a(n) = t n ® a for a E g, nEZ, we have the following commutation relations (a, bEg; m, n E Z):

[a(m), ben)] = [a, b](m + n) + mDrn,_n(alb)C,

[d, a(m)] = ma(m), [C, g] = o.

Affine Supemlgebras and Number Theory 435

We identify 9 with the sub algebra 1 0 9 of g. The bilinear form (.1.) extends from 9 to an even invariant bilinear form on g by:

(a(m)lb(n» = om,_n(alb), (qt, C 1)0 glCC + Cd) = 0,

(ClC) = (did) = 0, (Cld) = 1.

Let 6 ~ + cc + Cd be the Cartan subalgebra of g and let 11+ = n+ ffi

(EBj>o tj 0 ~). We identify 6* with 6 using the bilinear form (.1.)·

Given a subset S c ~, we let S = {a + nCla E S, n E Z} \ {O}. Then the set of roots of g with respect to 6 is Ll = ~o u {Or, the set of even (resp. odd) roots being Llo = ~o u {Or (resp. Lll = ~r). The set of positive roots of Ll associated to the choice of ~+ c ~ is Ll+ = ~+U{a+nCla E ~U{O}, n> O}, the set of simple roots being ft = {ao := C - O} UTI. As usual the subscript + means the intersection with Ll+, e.g., ~+ = ~ n Ll+.

Define p E 6* by P = p + hV d. One has:

As before, for a E ~o we let a V = 2a/(ala). For a E ~* define ta E Aut 6* by

ta(.~) = A + A(C)a - ((Ala) + ~(ala)A(C» C.

For an additive subgroup S c ~* the set ts := {tala E S} is a subgroup of Aut 6*. Let M(respMU) be the Z-span of the a V such that a E ~o (resp. a E ~~). The Weyl group of g is W = W rx: tM' The following subgroup of W is more important, however: WU = W rx: t Ma .

Denote by!ft the field of merom orphic functions in the domain Y = {h E 61 Re(Clh) > O}. Since C is W-fixed, the group W acts on !ft. Let q = e-c ; note that Iql < 1 in Y. Given A E ~*, we extend it to a linear function :x E 6* by letting :X(C) = :Xed) = 0; this gives us an embedding of the field F of rational exponential functions on ~ in the field !ft.

Let R be the Weyl denominator of g (see Section 2). Introduce the affine denominator by

00

R = R IT (1- qn)f IT (1 - qnea ) IT (1 + qnea)-l. n=l aELi.o

It is clear that R E!ft. As in Section 2, it is easy to check that

w(ePR) = E(w)ePR, wE W. (4.1)

-Here E : W --t ±1 is a homomorphism which extends the homomorphism E : W --t ±1 by E(tM) = 1.

436 V.G. Kac and M. Wakimoto

Theorem 4.1. Suppose that h v =1= 0. Then

ePR = L t,,(ePR). (4.2) "EM.

The proof of Theorem 4.1 is based on the following

Lemma 4.1. Suppose that A E 6* is such that (A + fJ)(C) =1= 0, and suppose that L(> .. ), A E 6*, is an irreducible subquotient of the Verma g-module M(A). Then there exists a sequence AO = A, Al, ... , Ak = A such that Ai is obtained from Ai-l (0 < i :$ k) in one of the following two ways:

(i) Ai = Ai-l - nf3i' where f3i E Lio+ and (Ai-l + fJlf3n = n E N, (ii) Ai = Ai-l - f3i, where f3i E Li H , (,Bdf3i) = 0, and (Ai-l + fJlf3i) = 0.

Proof. See [K6, Theorem 1b]. (Note that the usual proof is based on the determinant formula given by [K6, Theorem 1a]; the exponents in this formula should be corrected as in [KS]). •

A detailed proof of Theorem 4.1 will be given elsewhere. In the case def 9 = 1 the same proof works as that in Section 2.

Theorem 4.2. Let 9 be a simple finite-dimensional Lie supemlgebm with a non-degenemte Killing form, i.e., hV =1= 0. Choose a set of positive roots ~+ such that the set of simple roots contains a maximal set of pairwise orthogonal isotropic roots {f31," ., f3s}(s = def g). Let 13 = 'L,:=1f3i and let d, = dim g" f = ° or 1. Then

(4.3) where

Proof. From Theorems 2.1 and 4.1, and Remark 3.2c we have:

Assuming that IXj I are sufficiently small we expand in a geometric progression for each a:

Affine Supemlgebms and Number Theory 437

Hence we may rewrite (4.5) as follows:

Applying the specialization Ftpo to both sides and tending t to 0 we obtain:

The theorem now follows from (3.7). • Theorem 4.3. Let 9 be a simple finite-dimensional Lie superalgebra of type A(m, n), m i- n, or C(m). Then in the notation of Theorem 4.2 one has:

where r is the number of roots from.6.1+ which are negative with respect to the standard set of positive roots.

Proof. Let>. E ~* be such that (>'Ia) = 0 if a E .6.0 and (>'Ia) = ~ if a is odd and positive with respect to the standard set of positive roots. Now apply to both sides of (4.6) the specialization Ftpo-(Jogq)>. and tend t to O. •

Example 4.1. Let 9 = 81(2,1). Choose.6.+ such that both simple roots a1 and a2 are odd. Then (a1Iat) = (a2Ia2) = 0, (a1Ia2) = 1, P = 0, hV = 1, M" = Z(a1 + a2). We let x = e-<>l, y = e-<>2. Then (2.4) becomes (if we take (J = a1 or a2 respectively):

1-xy R : = -:-( 1-+-x"""')-:-( 1-=-+-y""7")

1 y -----1+x 1+y

1 x 1+y-1+x'

438 V. C. K ac and M. Wakimoto

Formula (4.2) for Ii becomes after expanding the right-hand side (for which we assume Iql < lxi, Iyl < 1):

00 (1- qn)2(1_ xyqn-l)(I_ x-1y-lqn) 11 (1 + xqn-l)(1 + x-lqn)(1 + yqn-l )(1 + y-lqn)

(m~o -m,~-J (_1)m+nxmynqmn).

(4.8)

This truly remarkable identity may be viewed also as the denominator identity for N = 4 superconformal algebra. An equivalent form of (4.8) is (here we assume Iql < Ixl < 1):

Replacing in this formula q by q2, X by z and y by qz-l, we obtain

This identity may be viewed as the denominator identity for N = 2 super­conformal algebra and also of the affine superalgebra gl(l, 1)". It also has been rediscovered many times (see, e.g., [KP]). It is worth mentioning here that Gauss identities (5.1) and (5.2) are denominator identities for N = 1 superconformal algebras.

Yet another useful form of (4.8) is obtained by dividing both sides by R:

00 (1 _ qn)2(1 _ xyqn)(1 _ x-1y-lqn) 11 (1 + xqn)(l + x-lqn)(l + yqn)(l + y-lqn)

= 1+ (l+x)(l+y) f (_lr+nqmn(xmyn_x-my-n). (1 - xy) m,n=l

(4.9)

Letting x = y = z, we obtain:

(4.10)

Example 4.2. Let g = osp(3, 2). Choose 6.+ such that both simple roots al and a2 are odd; one has: (alla2) = 0, (a2Ia2) = ~, (alla2) = -~, p = -~al'

Affine Supemlgebms and Number Theory 439

R'- (1 - y)(1 - xy) __ 1_ _ Y .- (1 +x)(1+ xy2) - l+x l+ xy2'

Formula (4.2) for Ii becomes after expanding the right-hand side (as before,

Iql < lxi, Iyl < 1):

00 (1- qn)2(1_ xyqn-I)(I_ x-Iy-Iqn)(I_ y2qn-I)(I_ y-2qn)

11. (1 + xqn-I)(1 + x-Iqn)(1 + yqn-l )(1 + y-Iqn)(1 + xy2qn-I)(1 + x- Iy-2qn)

= (m~o -m,~-J (_x)n (y-m - ym+2n+1) q!m(m+2n+1).

(4.11)

Here is a general construction of a specialization (el [Kg]). Let a be an automorphism of order m of {I such that a(h) = h for h E ~, let {I = E9jEZ/ m z {lE,j (10 = 0 or 1) be the corresponding Z/mZ--gradations and let dE,j = dim {lE,j' For 0: E Ll we have {Ie> C {IE,S", j let Se> E Z be such that o ~ Be> < m, se> = Se> mod m. Define A E ~ by (Alo:) = Se>' Replace q by qm in (4.2), divide both sides of it by n "'EAo+ (1 - e-e» and apply to the resulting

("1",)=0

identity the specialization Ft7 := F-(iOgq)A' The left-hand side becomes

II (1 - qj)dO,j /(1 + qj)d"j.

JEN

One can generalize the trick applied in the proof of Theorems 4.2 and 4.3 to evaluate the right-hand side (as in [Kg]). However, the answer is simpler when A is fixed under the action of W (as in Theorems 4.2 and 4.3). Another situation when this trick may be applied is when Se> = 0 for 0: E Ll~ and Se> =I 0 for 0: E Llo \ Ll~, so that A is fixed under W~.

Example 4.3. (a) Let a be the automorphism of {I such that ai llo 1, al ll, = -1. Then applying Ft7 to R/~ gives an expansion of

II (1 - q2j )dO /(1- q2j -1 )d l ,

JEN

which is Ll(q)dO (see (5.2) below) provided that do = d l .

(b) Let {I = A(n, n - 1) and choose Ll+ such that n consists of odd roots {0:1, ... ,0:2n} (see Example 1.1). Let a be such that Se>2j = 1 and Se>2j_1 = 0 (j = 1, ... , n). Then Ft7 gives an expansion of O(q)2n over an n-dimensional lattice. Namely, using the same method as that in the proof of Theorem 4.2,

440 V.G. Kac and M. Wakimoto

we obtain:

O( _q)2n = 2n L L( _1)l s lq.!!.:}!(olo)+(olpo)-(ols)

oEM. sES

n -1 X II (1 + q(n+1)(01,Bj)-(sl,Bj») ,

j=l

where § is the set of all subsets in ~o+, and lsi (resp. s) stands for the number (resp. sum) of elements of s E §. Note that M" is an n-dimensional lattice (of type An), so that for n = 1 we recover Jacobi's formula (0.1), but for n = 2, 3 and 4 our formula is more complicated than Jacobi's (0.2-4). The way to recover Jacobi's formulas will be explained below.

(c) Let g = A(2n + 1, 2n) = sl(2n + 2, 2n + 1) and let a be the conjugation by the matrix diag(In +l, -In+!; In+! , -In). Then a is an order 2 automorphism of g such that do,o = dO,l = d1,o = d1,1 = 2(n + 1)(2n + 1). Hence F" gives an expansion of O(q)2(n+!)(2n+!) over an n + I-dimensional lattice (n = 0, 1, ... ).

Example 4.4. Let g = A(m, n) with m > n. We choose the set of simple roots as in Example 1.1. Then formula (4.2) can be written, using (2.4) and (3.4), as follows:

( 4.12)

where M" is the lattice spanned over Z by elements O'2j-1 +O'2j (j = 1, ... ,n+ 1) and O'i (2n + 3 ::; i ::; m + n + 1). Introduce the element ~ E ~ by relations (~IO'j) = (-I)j /2 for 1 ::; j ::; 2(n + 1), WO'j) = 0 for 2n + 3::; j ::; m + n + 1. Then (~IO') = 0 for all 0' E D.o, hence tf. commutes with W. Applying tf. to both sides of (4.12) we obtain (assuming Iql < 1):

ep+,Bih = II L 1 (n+1 ( 00

(n+1)! j=l k3 ,r3 =O

• v L:n+l (4 13) " "() p-h 0- . rj02j_1 . X ~ ~ (; W we 3=1

k n +2 , ••• ,km EZ wEW

12hv (010)+ ",~+1 rj (kj+~)-"'':"' kj-1 (n+l)(2m-n) X q W3=1 W3=n+2 .

Here

Affine Supemlgebms and Number Theory 441

(J is the sum of the 'Y E DoH such that hl~) = ~,

n+1 m

0'= L kj (a2j-1 + ... + a2n+2) + L kjan+l+j j=l j=n+2

(so that (pia) = Ej=n+2 kj ). Dividing both sides of (4.13) by Ro and letting eOl, 1-+ 1 (i = 1, ... , m + n + 1), we obtain (using the Weyl dimension formula)

00 ( -)dO ( - 1)d1 a power series expansion for the product I1j =l 1 - qJ / 1 - qJ-, . Fur-

thermore, applying L{ to both sides of (4.13), we obtain a better expression for the affine denominator:

fJ-hva-E. rj Q 2j_l n1. L e JEJ IhV(OIIOI)+'"' r-k--'"' k-X €(w)w q2 L...J;EJ J J L...J;=n+2 J. n - (1 + e-OI2;-1) wEW JEJC

(4.14) Here

Sj = { (k, f) jEJ E Z21JII kj > 0 and rj ~ 0, or k j < 0, rj < 0 for each j E J} ,

€J(k, f) = (_I)H)EJ1kj<O} and JC = {I, 2, . .. , n + I} \ J.

Note that (p-hVa-EjEJrja2j-lla2i-l) = 0 if i E Je. Hence, by di­

viding both sides of (4.14) by Ro and letting eOl' 1-+ 1 (i = 1, ... ,m + n + 1), we may use Theorem 3.2 to obtain a power series expansion for

n~l (1 - qj)dO / (I + qj)d1 •

Example 4.5. Let g = B(m, m). We choose a set of simple roots al, .. . ,a2m E Dol with the following scalar products: (adai) = 0 for i = 1, ... ,2m-I, (a2mla2m) = -1/2, (ailai+l) = (_I)i+l /2 for i = 1, ... ,2m-I. We have: h V = 1/2 and M# is spanned over Z by vectors 2 (a2j-1 + a2j),

j = 1, . .. , m. Then formula (4.2) can be written using (2.4) and (3.4), as follows:

(4.15)

Introduce the element ~ E ~ by relations Waj) = (-I)j /2 for j = 1, ... , m -1, (~la2m) = o. Then w(~) - ~ E M# for all w E W. Applying t( to both sides of

442 V.G. Kac and M. Wakimoto

(4.15) we obtain a more convenient for specializations formula:

where f3

x E f(w)wi+!<€-a)- L:;:'l rj a 2j_l

wEW

X q!(a1a)+ L:~=l (kj(rj+lH!rj )-!m(m-l),

5. Applications to sums of squares and sums of

triangular numbers

Consider the following power series in q (that converge for Iql < 1):

D() '" n2 A(q) = '" qn(n+l)/2. q =L...Jq ,u L...J

(4.16)

One has the following product expansions due to Gauss (which follow imme­diately from the Jacobi triple product identity):

1- qk D(-q) = II 1 +qk'

k~l

1- q2k ~(q) = II 1 _ q2k-l .

k~l

(5.1)

(5.2)

Let ~j = j(j+1)/2 and let ~ = {~jlj E Z+} be the set of triangular numbers. Let DN,k = I{ii E ZNI L:f=l n; = k}1 (resp. ~N,k = I{ii E ~NI L:f=l nj = k}1) be the number of representations of k as a sum of N squares (resp. triangular numbers). Let DN(q) = L:kEZ+ DN,kqk and ~N(q) = L:kEZ+ ~N,kqk be the generating series for these numbers. Then, clearly

Thus, in view of (5.1) and (5.2) we may use the results of Section 4 to obtain expansion for DN(q) and ~N(q). In fact, formulas (4.3) and (4.7) give such expansions provided that do = d1 , which happens for g of type A(n, n - 1), B(n, n) or D(n + 1, n + 1). These gives us formulas for DN(q) and ~N(q), where respectively

N = 2n(n + 1), 2n(2n + 1) or 4(n + 1)2 (n EN).

Affine Supemlgebms and Number Theory

Example 5.1. 9 = A(l,O). Letting z --+ 1 in (4.10) we get:

00

O( _q)4 = 1- 4 E (_l)m+n(m + n)qmn m,n=l

00

= 1+8 E (_l)m+n-lmqmn m,n=l

= 1 +8 f: m(_q)m. m=l 1 + qm

. 2m~m 2m~m 4m<m . Smce l+q m = l-q m - l-q m' we obtam:

4 kqk O(q) = 1+8E~,

k>1 q 4!k

which is Jacobi's formula (0.2). In other words, for n ;::: 1:

04,n =8Ek. kl" 4tk

443

Letting z = i in (4.9), we get Jacobi's formula (0.1), i.e., that 02,n equals 4 times the difference between the number of divisors of n congruent to 1 mod 4 and the number of divisors of n congruent to -1 mod 4. This result goes back to Gauss.

Letting in (4.9) x = -y = ql/2, we obtain:

00

~(q)2 = E (_1)kq «2H l)(2k+l)-1)/4.

j,k=O

This means that ~2,n equals the difference between the number of divisors of 4n + 1 congruent to 1 mod 4 and the number of divisors of 4n + 1 congudent to --1 mod 4. Replacing in (4.9) q by q2, letting x = -qz, y = -q and tending z to 0, we get

00

~(q)4 = E (2k + 1)q((2H l)(2k+1)-1)/2.

j,k=O

This means that ~4,n equals the sum of all divisors of 2n + 1. This result is due to Legendre (1828).

444 V.C. Kac and M. Wakimoto

Example 5.2. 9 = B(l, 1). Dividing both sides of (4.11) by (1 - xy)(l - y) and letting x = y = Z -4 1, we obtain Jacobi's formula (0.3). In other words,

D6•n =4 L (_1)!(j+1)(j2_4k2). j,k2!l

j odd,jk=n

Furthermore, replacing in (4.11) q by q2 and letting x = y = -qz, we obtain after dividing both sides by (z - 1)2 and letting z -4 1:

1 ~(q)6 = __

8 L (_1)W- 1)(k+1)(j2 - k2)qWk-1).

j,k odd j>k;;::l

!(j-k) odd

In other words, as can be easily seen:

~6.n=~ L (_1)!U+1) (j2_k2). j>k?:l

jk=4n+3

Example 5.3. 9 = A(m,m - 1). Dividing both sides of (4.13) by Ro, letting ea , -4 1 (i = 1, ... ,2m) and replacing q! by -q, we obtain

where

(5.3)

Similarly, we obtain from (4.14) using the remarks and notation of Example 4.4:

O( _q)2m(m+1)

(-1) !m(m+1)2m

nm (")2 j=l J.

x 2- IJc1c C~ ri + IJcl- 1, m2 + IJcl- 1) il (rj + k1 + ... + km)

x II (ri - rj) (r~ - rj) q'L-l$'$j$m k,kj+ 'L- jEJ kjrj

l~i<j~m

Affine Superalgebras and Number Theory 445

Example 5.4. 9 = B(m, m). Dividing both sides of (4.16) by Ro, letting ea , ~ 1 (i = 1, ... , 2m) and replacing q! by -q we obtain (using the Weyl dimension formula):

6.(q)2m(2m+1)

= 2:m! ( L k.rEZ't

) m (l)k.

- j 1 + 2r 1 + r' ~ L IT «2' _ 1)!)2 ( j)( j) k.rE(-N)'" J=1 J

x IT l$i<j$m

"'." (k~+2rjkj+2kj+rj)-!m(m-l) X qWj=' J •

Conjecture 5.1.

6.(q)2m(2m+1) = 1 2 L (_l)lkl m! (n;1(2j -I)!) k.rEZ't

x IT (ri - rj)(r~ - rj) (2 + r~ + rj) (1 + ri + rj) l$i<j$m m

X IT (1 + 2rj) (1 + rj) q2:::, «k~+2kj )+(rj(2kj+1» )-!m(m-1).

j=l

6. Some "degenerate" theta functions and modular forms

In this section we use the denominator identity (4.8) for 8l(2, 1)" to derive transformation properties of certain "degenerate" theta functions and modular forms, and related to them characters of N = 2 superconformal algebras.

Let q = e 27rir where Imr > 0, and let 8(T, z) be the standard Jacobi theta function:

8(r,z) = e7rir/4e-7riz IT (1- qn) (1- qne-27riZ) (1- qn-le27riZ)

n2:1

= L( _1)jqW-!)2 e 27ri(i-!)z. jEZ

Recall that this theta function has the following transformation property (see, e.g. [K9)):

446 V.C. Kac and M. Wakimoto

Introduce the following function in 1', Zl, and Z2:

F( ) _ 1](1')38(1', Zl + Z2) 1',Zl,Z2 - 8( )8( )' 1',Zl 1',Z2

where 1](1') is the Dedekind 1]-function. Note that F(1', Zl, Z2) is the left-hand side of identity (4.8) where x = _e211"iz1 , y = _e211"iz ••

Fix a positive integer m. Using the transformation formula for 8 and identity (4.8), it is not hard to derive the following transformation formula:

( m Zl Z2) l' .,,101 '2 L 2"i(a'l +b'2) ab F --,-,- =-e m... e m qmF(m1',Zl+a1',Z2+ln').

l' l' l' m a,bEZ!mZ

(6.1) Given j, k E !Z, introduce the following functions:

(m)± 2.!!. ."i(j-k). ( • 1 1 1 1) Fj,k (1',z)=q m e m F m1',-z+J1'-4±4,z+k1'+4T 4 .

We call them degenerate theta functions since, due to (4.8), they have the following power series expansion associated to a bilinear form representing zero:

F~':)±(1', z) = (t ] , a,b=O

.,~,) (±! )'-. qm( .+,t,)( >+,.) " •• « >+;. )-"+"»'.

(6.2) Transformation formula (6.1) implies the transformation formulas for in­

definite theta functions. In order to state them introduce the following sets:

A~={(j,k)EZ210<j,j+k<m, O:::;k<m},

A; = {(j, k) E G + Z)210 < j, k,j + k < m}. Introduce the following numbers:

(m) 2 "i(j-k)(a-b) • 7r(j + k)(a + b) S('k)( b) = -e m sIn . 3, a , m m

Provided that j, k E Z, one has:

(m)± ( 1 z) . _a.n.u.! '"' (m) (m)+() Fj,k -:;:. :;: = -Z1'e m... ~ S(j,k)(a,b) Fa,b 1', Z . (a,b)EA;!;

Provided that j, k E ! + Z, one has:

(m)± ( 1 z) . _ hi.· '"' SCm) F(m)- ( ) Fj,k -:;:' :;: = -Z1'e m... ~ (j,k)(a,b) a,b 1', Z •

(a,b)EA;!;

(6.3)

(6.4)

Affine Superalgebras and Number Theory 447

It follows that the linear span offunctions {Fj,r;)+ and F;,r;)-I(j, k) E A;t.UA;;.} is invariant with respect to the action of SL2(Z) given by

1 hie.' (aT + b z) F(T,Z)I(4b)=(cr+d)-e m (eT+d)F --d'--d . ed CT+ cr+

We want now to specialize to z = O. Let <t';~)+(T) = F;,r;)+(T,O) for all

(j,k) E A;;. and <t';~)-(T) = Fj~r;)-(T,O) for all (j,k) E A;;'UA;t. except for (j,O) E A;t. (when the series diverges). These functions have the following power series expansions:

In order to get SL2(Z)-invariance, we must add functions <t';:;;)-(T) (j E Z, o < j < m) given by

Then we have:

(m)+ (1) . '" S (m)-( ) 'f(' k) A-<t'j,k -:;: = -ZT L...,; (j,k)(a,b) <t'a,b T 1 J, E m'

(a,b)EA;t;

(m)- (_!) __ . '" S. (m)-(resp +)( ) 'f ( . k) E A-(resp+) <t'j,k T - ZT L...,; (],k)(a,b) <t'a,b T 1 J, m •

(a,b)EA;;'

Note also that

(m)+ 2,,'jk (m)- .. _ <t'j,k (T + 1) = e m <t'j,k (T) If (J, k) E Am,

<t';~)-(T + 1) = e ''';''jk <t';~)+(resP-)(T) if (j,k) E A;.(resp+) .

Thus we have constructed a finite set of modular forms of weight 1 whose linear span is SL2(Z)-invariant.

We derive now, using (6.3) and (6.4) the transformation properties of char­acters and supercharacters of the N = 2 superalgebras (cf. [RY]). Introduce

448 V.G. Kac and M. Wakimoto

the following normalized N = 2 demoninators:

From the transformation formula for e and 11 one gets:

q,(O)- ( _ ~, ~) = Te- ,,~2 q,(O)- (T, z),

w(OH (_~,~) = _iTe-,,~2 q,(1/2)-(T,Z),

q,(1/2)- (_~,~) = _iTe-,,~2 q,(O)+(T,Z),

q,(1/2H ( 1 z) _ . _ ",.2 q,(1/2)+ ( ) --:;:' -:;: - -ZTe T T, Z ,

One knows that the normalized characters and supercharacters of the uni­tary discrete series of the two N = 2 superalgebras corresponding to E = 0 or ! of central charge 3cm , where Cm = 1 - ~, m = 2,3, ... , are given by the following formulas ([Dj,[Mj,[RYj):

p(m)-( ) (m,E) _ j,k T, Z

chj,k (T,Z) - q,(EH(T,Z) ,

p(mH( ) (m,E) _ j,k T, Z

schj,k (T,Z) - q,(E)-(T,Z) '

if j, k E E + Z; the range of (j, k) is A;t or A;;;, depending on whether E = 0 or !. From the above transformation formulas we obtain the following theorem.

Theorem 6.1. (cf. [RY]) (a) If(j,k) E A;;;" then

h (m,1/2) ( ! ~) _ ",.2 em ~ SCm) h(m,1/2)( ) C j,k - T' T - e T L.... (j,k)(a,b) C a,b T, Z ,

a,bEA;;;

h(m,1/2) ( 1 z) _ 1!.l.C Cm ~ SCm) h(m,O)() sc j,k --:;:' -:;: - e T L.... (j,k)(a,b) C a,b T, Z ,

a,bEA;;'

(m 1/2) 2"i; k ,,' (m 1/2) ch· k' (T+l,z)=e m -TSCh'k' (T,Z), 3, J,

(m 1/2) 2"ijk ,,' (m 1/2) sch'k' (T+l,z)=e m -TCh'k' (T,Z). J, 3,

Affine Superalgebms and Number Theory 449

(b) If (j, k) E A;t" then

hem 0) ( 1 z) " .. 2 c """" SCm) h(m,1/2) ( ) C j,k' -;:. -:;. = e T m L...J (j,k}(a,b) SC a,b 7, Z ,

a,bEA;;;

(m,O) ( 1 z) . ~Cm """" SCm) h(m,O) ( ) schj,k --:;.' -:;. = -te T L...J U,k}(a,b) SC a,b 7, Z ,

a,bEAt.

(m 0) 2"ijk (m 0) ) ch. k ' (7+1,z)=e m ch. k' (7,Z, 3, J,

(m 0) 2"ijk (m 0) ( ) sch. k' (7 + l,z) = e m sch. k' 7,Z. 3, ],

Corollary 6.1. The linear span of all normalized characters and supercharac­ters of the minimal unitary discrete series with central charge 3cm is invariant under the action of S L2 (Z) defined by

( a b) f( ) - ".c.2 C f (a7 + b z ) 'T,Z = e C'T+d m --, -- •

c d c7+d cr+d

Example 6.1. m = 2 (minimal pOSSible). Then At {(I,D)}, A2' {( 1 1)} d h(2,€) h(2,€) 1 Fu h 2' "2 ' an c j,k = SC j,k =. rt ermore,

+ ( (2)+ (1/2)-) * (!)2 1;'1/2,1/2(7)= F1/2,1/2(7,Z)=W (7,Z) z=o=q ~ q ,

_ _ ( (2)- _ (1/2)+ ) _ * ( !) 2 1;'1/2,1/2(7) - F1/2,1/2(7, z) - W (7, z) z=O - q ~ -q ,

-()_(F(2)-( )_,y.(o}+( ») _10( )2 1;'1,0 7 - 1,0 7, Z - 'J.' 7, Z z=O -"2 -q.

These three functions span a 3-dimensional SL2(Z)-invariant space of modular forms of weight 1. (Note that F:~J+ (7, z) has a pole at z = D.)

Remark 6.1. Similar results may be obtained for the affine denominator in the general case. As in [K5], this may be used in order to prove the "strange formula" (1.2).

7. The case hV = 0

The simplest non-trivial case when hV = 0 is g = A(I,I). It is more conve­nient to consider g = gl(2, 2) instead, with simple roots at, a2, a3, such that (a1Iat} = (a3Ia3) = 0, (a3Ia3) = 2, -(a1Ia2) = (a3Ia2) = 1. We conjecture that the denominator identity still holds with an extra factor. Namely, letting x = e-a1 , y = e-a2 , z = e-a3 , introduce the following functions (fixed under

450 V.G. Kac and M. Wakimoto

In = x-nz-n(x2n-1 - z2n-1)(x - z) L (_1)jqU+1)(H2n)/2.

jEZ+

Conjecture 7.1. ePR= (1+ tIn) Ltna2 (ePR). n=l nEZ

If we divide both sides by R and let x, y, z -+ 1 we obtain Jacobi's formula (0.4), i. e.,

Ds.n = 16L(-1)n+j j3. jJn

So far we have entirely left out the series of simple Lie superalgebras g of type Q(m), which do not admit a non-degenerate even invariant bilinear form (but admit an odd one), hence it is natural to let hV = O. Its even part go is a simple Lie algebra of type Am and the go-module gl is the adjoint Am-module. Let A (resp A+) denote the set of roots (resp positive roots) of Am, let n = {01,"" am} be the set of simple roots, let p be the half of the sum of roots from A+, let W be the Weyl group, and let (.1.) be the W-invariant bilinear form normalized by (oiloi) = 2. Let s = [mt1] and consider the following (maximal orthogonal) set of s roots: /1 = 01 + .. '+Om, /2 = 02 + ... + 0m-1, .... The denominator identity for Q(m) is given by the following formula:

The associated to Q(m) affine superalgebra 9 is a central extension (by CC) of the Lie superalgebra E9jEz tj gjrnOd2 (its Cartan matrix is not sym­metrizable). We let q = _e-c and define the affine denominator as follows:

In order to state the affine denominator identity, define ta for 0 E ZA as follows

Affine Supemlgebras and Number Theory 451

Conjecture 7.2.

L:: €( W )q1k1wtnl"Yl + ... +n.')'. (eP+ E, k,')',) . wEW nEZ+ kEZ+

kl~···~k"

In particular, for m = 1, Conjecture 7.2 reads

where x = e-a1 • This identity follows from (4.8) by replacing q by q2, Y by -qx and x by -q. As we mentioned in Section 5, dividing both sides by 1 - x and tending x to 1, we recover Legendre's formula b.(q)4.

For m = 2, Conjecture 7.2 reads (here x = e-a1 , y = e-O/2 ):

00 (1 _ q2n)2(1 _ q2n-2x )(1 _ q2n-2y)(1 _ q2n-2xy) !! (1- q2n-l)2(1_ q2n-l x)(l_ q2n-ly)(I_ q2n- l xy)

(1 - q2nx -l )(1 - q2ny-l )(1 _ q2nx-l y-l) X~~~~--~~--~~~~~~~~~~7 (1 - q2n- l x l)(l _ q2n ly-l)(1 _ q2n lx ly l)

= L:: L:: qik-l(xy?-i(l_ xi)(1 - yi)(I- xiyi). jEN kENodd

Letting in this identity x = y = z, we obtain after dividing by (1- z)2(1- z2):

00 (1- q2n)2(1_ q2nz )2(1_ q2nz -l)2(1_ q2nz2)(1_ q2nz -2) !! (1- q2n-l)2(1_ q2n-lz)2(1_ q2n-lz-l)2(1_ q2n-lz2)(1 _ q2n-lz-2)

L:: L:: 'k-l 2-2' Z3 - 1 z 3 - 1 ( . )2 2' = qJ Z J __ _ __ .

z -1 Z2 - 1 jEN kENodd

Letting z tend to 1, we obtain another Legendre's formula:

b.(q)S = L:: L:: j3qik-l, jEN kENodd

i.e., b.s,n = L:: (n; 1) 3

kln+l kodd

Finally, dividing both sides of the identity for general m by R and evaluat­ing at 0 using Weyl's character formula we obtain the following two beautiful results for m = 28 - 1 (resp. 28):

b.4s2,n(respb.4s(s+l),n) = L:: L:: dimL(klll+···+ksIs,Am), (7.2) f'EZ+ kEFr.n

452 V. C. K ac and M. Wakimoto

where

Fr,n = {k E Z+lkl 2: k2 2: ... 2: k8' t(ki(2ri + 1) + (m + 2 - 2i)ri) = n} , ,=1

and L(A, Am) denote the irreducible (finite-dimensional) module over Am with highest weight A.

The simplest new formula corresponds to m = 3:

1 D.16 =-,n 12 L (2kl +3)(2k2+ l)(k1 -k2+ 1)2(k1 +k2+2)2.

rl,r2 EZ+ kl2:k2EZ+

kl (2rl +ll+k2(2rl +1)+3q +r2=n

Letting a = 2k1 + 3, b = 2k2 + 1, r = 2r1 + 1, s = 2r2 + 1, we get the following beautiful form of this formula:

D.16,n = 3 \3 L ab (a2 - b2)2. a ,b,r,BENodd

,,>b ar+bs=2n+4

The next new formula corresponds to m = 4:

L a3b3 (a2 _ b2 )2.

a,bEN r,sENodd

,,>b ar+bB=n+3

In a similar way the general formula (7.2) can be rewritten as follows:

4-s(s-1) D. 2 - ---,:--.---

48 ,n - n28- 1 ., j=l J.

L a1 ... as II (a~ - aJ)2 0lo· ... ClsENodd i<j rl.···,rs ENOdd

41>···>48 a1 rl + ... +asTs=2n+s 2

L (a1 . . ' as )3 II (a~ - aJ)2 . 01 t'·· ,as EN i<j

rt •... ,rs EN odd °1>···>a6

at rl + ... +osrs=n+!s(s+l)

Affine Supemlgebras and Number Theory 453

8. Integrable highest weight modules over 9

Given a choice of ~+ C ~, we associate to each A E 6* an irreducible highest weight module L(A) over 9 defined by the property that there exists a non-zero vector VA such that:

h(VA) = A(h)vA for h E 6, n+(VA) = 0, (tn ® g)VA = 0 for n > O.

The number c = A(C) is called the level of L(A). Note that L(A) := U(g)VA is an irreducible highest weight module over g.

Definition 8.1. A g-module L(A) is called integrable if the following two properties hold:

(i) dim L(A) < 00,

(ii) tn ® go are locally nilpotent for all a E ~~ and n E Z.

It is clear that if L(A) is an integrable highest weight g-module for some choice of ~+, then it is an integrable highest weight module for any other choice of ~+; also, L(A) is g-locally finite. The g-module L(A) is integrable if and only if chL(A) is invariant with respect to W ~ tMI.

In order to classify integrable highest weight g-modules L(A), choose ~+ such that II has a unique odd root (cf. Example 3.3). Note that ao is even if and only if 9 is of the second type; in this case define e E ~+ c .6.+ as in Example 3.3. If 9 is of the first kind, i.e., of type A(m, n) or C(n), we define e = ao + ii', where the coefficients of ii' in terms of II are the following:

A(m,n), m 2 n C(m), m> 2

O-O···-O-®-Ol-···-Ol ®l_ 0- 0 - ... - 0 {= O.

As before, we let k = (AleV ) and ki = (Alan for all non-isotropic ai.

Theorem 8.1. (a) For 9 of the second kind, the g-module L(A) is integrable if and only if dimL(A) < 00 (the corresponding conditions on k and the ki' i =I- 0, are given in [K2, p. 84J) and ko E Z+.

(b) For 9 of type A(m, n), m 2 n, the g-module L(A) is integrable if and only if

(i) k E Z+ and ki E Z+ for i =I- 0 and i =I- m + 1; (ii) when k :::; n, there exist r, s E Z+ such that: r + s = k, (Alao) =

r+km +n+1 +km +n+·· ·+km +n- r+2 , (Alam +1) = s+km +2+km +3+·· ·+km +s+1'

The C(m)-module L(A) is integrable if and only if (i) k E Z+ and ki E Z+ fori =I- 0,1,

(ii) if k = 0, then (Alao) = (Alat) = O.

Proof. We use two facts: (i) if a is a simple even root, then the root vec­tor attached to -a is locally nilpotent on L(A) if and only if (AlaV ) E Z+;

454 V.G. Kac and M. Wakimoto

(ii) any root from Li can be made simple by applying a sequence of odd simple reflections. •

One has the following expressions for the affine central charge c:

A(m,n), m 2:: n: C(m) : B(m,n), D(m,n): D(2, 1; a), F( 4), G(3):

c = kl + ... + km + k. c = k2 + ... + km + k. c = ko + kl + ... + kn - l + k. c = ko + k.

It follows that the affine central charge c of an integrable g-module L(A) is a non-negative integer and that c = 0 if and only if the g-module L(A) is I-dimensional (which happens if and only if AIHcc = 0).

Example 8.1. g = ..4.(1,0). We choose II = {al,a2} where both al and a2 are odd. Then (allal) = (a2Ia2) = 0, (alla2) = 1. Then ao = C - al - a2 is an even root. Let mi = (Alai), i = 0,1,2. One has dim L(A) < 00 if and only if ml + m2 EN or ml = m2 = 0, and L(A) is integrable if, in addition, mo E Z+. The central charge c = mo + ml + m2 (E Z+). Assume that the following "non-degeneracy" condition holds:

mi =1= 0 (i = 1 or 2) => mi is not divisible by c + 1.

Then the same argument as in Section 2 gives the following character formula:

ePRchL(A) = :E tc< (ePRchL(A»). c<EMI

In particular, if ml = m2 = 0, we obtain for c E Z+:

ePRchL(cd) = :E tc< (ep+cdR). c<EMI

Letting x = e-C<l, y = e-C<2 we may rewrite (8.2) as follows:

i,kEZ+ min(i,k)E(c+l)Z

i,kEN m;n(j,k)E(c+l)N

(8.1)

(8.2)

(8.3)

Definition 8.2. An integrable g-module L(A) is called tame if its character is given by formula (8.1).

Theorem 4.1 says that the I-dimensional g-module is tame if h v =1= O. More generally, it seems that an integrable g-module L(A) is tame if c+ hV =1= 0 and

Affine Supemlgebms and Number Theory 455

(A+pla) is either 0 or is not divisible by c+hv for any a E .6.1; in particular, £(cd) is tame for c E N (it is clear that this condition is necessary).

References

[BL] I. N. Bernstein and D.A. Leites, Character formulae for irreducible representations of Lie superalgebras of series gl and sl, C.R. A cad. Bulg. Sci 33 (1980), 1049-51.

[D] V.K. Dobrev, Characters of the unitarizable highest weight modules over the N = 2 superconformal algebras, Phys. Lett. 186B (1987), 43-51.

[F] N.J. Fine, Some basic hyper geometric series and applications, Math. Surveys 27, 1988, AMS, Providence.

[H] D. Hickerson, A proof of the mock theta conjectures, Invent. Math. 94 (1988), 639-660.

P] C.G.J. Jacobi, Fundamenta nova theoriae functionum ellipticarum, Crelle J. (1829), 55-239.

PHKTl] J. Van der Jeugt, J.W.B. Hughes, R.C. King and J. Thierry-Mieg, Character formulae for irreducible modules of the Lie superalgebra sl(m/n), J. Math. Phys. 31 (1990), 2278-.

PHKT2] __ , A character formula for singly atypical modules of the Lie su­peralgebra sl(m/n), Comm. Aig. 18 (1990), 3453-3480.

[Kl] V.G. Kac, Infinite-dimensional Lie algebras and Dedekind's 'f/­function, Funct. Analis i ego Prilozh. 8 (1974), No.1, 77-78. English translation: Funct. Anal. Appl. 8 (1974), 68-70.

[K2] __ , Lie superalgebras, Advances in Math. 26, No.1 (1977),8-96. [K3] __ , Characters of typical representations of classical Lie superalge­

bras, Comm. Aig. 5 (1977),889-897. [K4] __ , Representations of classical Lie superalgebras, Lect. Notes

Math. 676, Springer-Verlag (1978), 597-626. [K5] __ , Infinite-dimensional algebras, Dedekind's 'f/-function, classical

Mobius function and the very strange formula, Advances in Math. 30 (1978), 85-136.

[K6] __ , Contravariant form for infinite-dimensional Lie algebras and superalgebras, in Lecture Notes in Physics 94, Springer-Verlag (1979) 441-445.

[K7] __ , Laplace operators of infinite-dimensional Lie algebras and theta functions, Pmc. Natl. Acad. Sci. USA 81 (1984), 645-647.

[K8] __ , Highest weight representations of conformal current algebras, Symposium on Topological and Geometrical methods in Field theory, Espoo, Finland, 1986. World. Sci., 1986, 3-16.

456 V.C. Kac and M. Wakimoto

[K9) __ , Infinite-dimensional Lie algrebras, 3rd edition, Cambridge Uni­versity Press, 1990.

[KL) V.G. Kac, J. van de Leur, Super boson-fermion correspondence, Ann. d'Institute Fourier 37 (1987), 99-137.

[KP) V.G. Kac, D.H. Peterson, Infinite-dimensional Lie algebras, theta functions and modular forms, Advances in Math. 53 (1984) 125-264.

[Ko) B. Kostant, On Macdonald's 17-function formula, the Laplacian and generalized exponents, Advances in Math. 20 (1976) 179-212.

[Mac) I.G. Macdonald, Affine root systems and Dedekind's 17-function, In­vent. Math. 15 (1972) 91-143.

[M) Y. Matsuo, Character formula of c < 1 unitary representation of N = 2 superconformal algebra, Prog. Theor. Phys. 77 (1987), 793-797.

[P) I. Penkov, Generic representation of classical Lie superalgebras and their localization, Monatshefte fur Math. (1994).

[PSI] I. Penkov, V. Serganova, Cohomology of G/ P for classical Lie super­groups and characters of some atypical G-modules, Ann. Inst. Fourier 39 (1989), 845-873.

[PS2] __ , Representations of classical Lie superalgebras of type I, Indag. Math N.S.3(4) (1992),419-466.

[RY) F. Ravanini, S.-K. Yang, Modular invariance in N = 2 superconformal field theories, Phys. Lett. 195B (1987), 202-208.

[S] V. Serganova, Kazhdan-Lusztig polynomials for the Lie superalgebras GL(m/n) Adv. Sov. Math. 16, part 2 (1993), 151-165.

[Se] A. Sergeev, Tensor algebra of the defining representation as the mod­ule over Lie superalgebras gl(m, n) and Q(n), Math. USSR, Sbornik 51 (1985), 419-424.

[TMl] J. Thierry-Mieg, Table des representations irreducibles des superal­gebras de Lie, unpublished (1983).

[TM2] __ , Irreducible representations of the basic classical Lie superalge­bras SU(m/n), SU(n/n)/ U(I), OSp(m/2n) , D(2/1, a), G(3) and F(4), Lect. Notes in Physics 201, eds. G. Denardo, G. Ghirardi and T. Weber, Springer, Berlin, 94-98.

[VJ1] J. Van der Jeugt, Irreducible representations of the exceptional Lie superalgebras D(2, 1 : a), J. Math. Phys. 26 (1985), 913-924.

[VJ2) __ , Character formulae for the Lie superalgebra C(n), Comm. Alg.

V. Kac, Department of Mathematics, M.LT., Cambridge, MA 02139

M. Wakimoto, Department of Mathematics, Mie University, Tsu 514, Japan

Received March 15, 1994

Quasi-Equivariant V-Modules, Equivariant Derived Category, and

Representations of Reductive Lie Groups

Masaki Kashiwara and Wilfried Schmid

1. Introduction

In this note, we describe proofs of certain conjectures on functorial, geomet­ric constructions of representations of a reductive Lie group GIR. Our meth­ods have applications beyond the conjectures themselves: unified proofs of the basic properties of the maximal and minimal globalizations of Harish­Chandra modules, and a criterion which insures that the solutions of a GIR-invariant system of linear differential equations constitute a represen­tation of finite length.

Let G be a connected, reductive, complex algebraic group, and GIR a real form of G - i.e., a closed subgroup of the complex Lie group Gan, whose Lie algebra gR lies as real form in the Lie algebra g of G. We fix a maximal compact subgroup KIR C GR; KR is the group of real points of some alge­braic subgroup KeG. As subgroups of G, GIR and K operate on the flag variety X of g. For each linear form A on the universal (Le., equipped with a positive root system) Cartan algebra, we consider the bounded equiv­ariant derived categories D~R'>" (X), D~,>. (X) of constructible sheaves of ((:>vector spaces on X with twist A, and the twisted sheaf of holomorphic functions OX(A). In [K2], one of us conjectured:

a) There exists a natural equivalence of categories «I> : D~ >. (X)~

D~R'>' (X); ,

b) the Ext-groups ExtP(S,OX(A», for S E D~R'>'(X), carry natural Frechet topologies and continuous linear GIR-actions;

c) the cohomology groups Hq(X,S ® OX(-A», for S E D~R'>'(X), have natural DNF topologies and continuous linear GR-actions;

d) the resulting representations ofGR on ExtP(S,OX(A» and Hq(X,S® Ox( -A» are admissible, of finite length;

e) ExtP(S, OX(A» and Hd-P(X, S®Ox( -A», with d = dimeX, are each other's strong duals;

f) if M E D~,>.. (X) is the image of a holonomic (V_>., K)-module VJt under the Riemann-Hilbert correspondence, then HP(X, VJt) coincides with the dual of the Harish-Chandra module of KIR-finite vectors in Extd-p(<<I>M,OX(A». (1.1)

W. Schmid was partially supported by NSF

458 M. K ashiwara and W. Schmid

The first of these conjectures was established by Mirkovic-Uzawa­Vilonen [MUV). In this note, we sketch proofs of (b-f).

In effect, (1.Ib-f) may be viewed as counterparts, on the level of group representations, of the Beilinson-Bernstein construction of Harish-Chandra modules. Our positive answer to the conjectures has already been used, in [8VI,2j, to establish other, related conjectures. A general discussion of these matters can be found in [KI, K4).

8pecial cases of (1.Ib-f) had been worked out previously. If j : D '---+ X is the inclusion of a GR-orbit D, and S = j,E the proper direct image of a GlR-equivariant twisted local system E on the orbit, then ExtP(j,£, OX(A» can be identified with an appropriately defined local cohomology group along D, and Hq(X,j,£ ® Ox( -A» with compactly supported cohomol­ogy of an appropriate sheaf on D. These representations are produced and studied in [HT,81,8W), and the remaining assertions (1.le,f), in these special cases, then follow from results in [HM8W, 82).

It turns out that ExtP(S,OX(A}} and Hq(X,S®OX(-A}} are, respec­tively, the maximal and minimal globalizations, in the sense of [82), of their underlying Harish-Chandra modules. It is not difficult to construct the maximal and minimal globalization functors. However, two crucial properties - topological exactness and the explicit characterization of the maximal and minimal globalizations of principal series representations -are not so obvious. The arguments outlined in [82) depend on relatively subtle lower bounds for the matrix coefficients of Harish-Chandra modules. Our proofs of the conjectures (1.Ib-f) not only depend on these proper­ties of the maximal and minimal globalization functors, they also imply them. As a consequence, we obtain alternate, more satisfactory proofs of the results announced in [82).

The first named author observed that the explicit characterization of the maximal globalization of principal series representations is but a spe­cial instance of a quite general phenomenon. Representations of a reductive Lie group OIR often arise as spaces of solutions of a system of GR-invariant, linear differential equations on a manifold on which GR acts. Intuitively, one expects the space of solutions of such a system to be a representation of finite length only if the manifold is "small" and the system of differential equations "strong." Theorem (7.6) below gives a precise sufficient condi­tion. In particular, the theorem covers the following situation. Let Z be a smooth, quasi-projective variety over C, with an algebraic action of the complexified group G, ZR a GR-invariant real form of Z, and JLz : T* Z -+ g* the moment map. We consider a G-invariant linear system of differential equations on Z, with algebraic coefficients, and let !)Jl denote the Vz­module defined by the system. The space of hyperfunction solutions of the

Equivariant Derived Category and Representations 459

restricted system on ZR has a natural Frechet topology, constitutes a GR-representation of finite length, and coincides with the maximal globaliza­tion of its space of KR-finite vectors, provided the following two conditions are satisfied:

a) VJt is annihilated by an ideal of finite codimension in Z(g) (= center of U(g»;

b) the characteristic variety Ch( VJt) intersects JLzl(b.l) in a Lagrangian subvariety of T* Z, for every Borel subalgebra beg. (1.2)

Condition b) holds vacuously if a Borel subgroup B of G acts on Z with only finitely many orbits - for example, if Z is a flag variety, or a quotient of G by a maximal unipotent subgroup U, or a complexified symmetric space G / K. In the case of a flag variety, even the "empty" system of differential equations satisfies a). In particular, theorem (7.6) applies to the principal series, to Whittaker models [GW, Mo], and to Helgason's conjecture for affine symmetric spaces [He, KMOT].

We exhibit the representations (1.1b,c) as cohomology groups of ob­jects in a derived category of representations. The construction of this derived category, in section three below, is of some independent interest. To make the connection between the Beilinson-Bernstein construction and GIR-representations, we set up a correspondence between GR-sheaves on X and certain 'V-modules on the complexified symmetric space G/K. Our arguments, and also the statement of Theorem (7.6), involve the notion of quasi-equivariant V-module, which was introduced, independently, in [K3] and - under the names "weakly equivariant 'V-module" and "weak (Vx, G)-module" - in [BBM] and [BB3]; we shall elaborate upon it in Section 4. This, too, has implications beyond the proofs outlined in this note. We intend to publish our results in more detailed form, including complete proofs, in the future.

We are pleased to dedicate this paper to Bertram Kostant on the oc­casion of his 65th birthday.

2. Minimal and maximal globalization

We consider the category R( GR) of GR-representations - by definition, its objects are complete, locally convex, Hausdorff topological vector spaces with continuous, linear GR-action, such that the resulting representation is admissible, 1 of finite length; morphisms in R(GR) are continuous, linear,

1 Le., in the restriction to KR, each irreducible representation of KR occurs only finitely often.

460 M. Kashiwara and W. Schmid

GR-invariant maps. For V E n(GR ),

VKR-/ini = linear span of the finite dimensional,

KR-invariant subspaces of V (2.1)

is dense in V and consists entirely of Coo vectors [HC3]. Both 9 and K operate on VKR-/ini - the former by differentiation of the GIR-action, the latter by complexification of the KR-action - and the two operations are compatible. Thus VKR-/ini becomes an algebraic (g,K)-module. On the infinitesimal level, the hypotheses of admissibility and finite length on the representation V imply that VKR-/ini is in fact a Harish-Chandra module: an algebraic (g, K)-module, finitely generated over the universal enveloping algebra U(g), with finite K-multiplicities [HC1,2].

We denote the category of Harish-Chandra modules by ,}{C(g, K). It is a full subcategory of Mod(g, K), the category of algebraic (g, K)-modules and (g, K)-invariant linear maps. The passage from V to VKR-Jini defines a functor

HC : n(GR) ~ ,}{C(g, K). (2.2)

This functor is faithful, exact, and assigns irreducible Harish-Chandra mod­ules to irreducible representations [HCl-3]. For ME ,}{C(g, K), the K-finite subspace of the algebraic dual,

M' = (M*)K-Jini' (2.3)

is another Harish-Chandra module, the Harish-Chandra module dual to M. We write V' for the continuous dual of a topological vector space V, and topologize V' with the strong dual topology. The natural GIR-action on the dual V' of some V E n( GR) need not be continuous; when it is continuous, then the resulting representation has finite length, is admissible, and

HC(V') ~ (HC(V»' , (2.4)

i.e., duality of Harish-Chandra modules corresponds to duality of represen­tations.

By a globalization of a Harish-Chandra module M, we shall mean a GR­representation V E n(GR) such that M = HC(V) . Every Harish-Chandra module can be globalized [C1,P], and this fact makes it a relatively simple matter to construct functorial globalizations, i.e., right inverses to the func­tor (2.2). In the next definition, we view COO(GR) as left (g, KR)-module via right translation, composed with the canonical antiautomorphism of

Equivariant Derived Category and Representations 461

{I, and as GJR-module via left translation. Every M E 1iC({I, K) has a countable vector space basis, so

(2.5)

inherits a Frechet topology and a continuous, linear GR-action from COO (GJR). The space MG(M) remains unchanged if one replaces Coo(GR) by the space of real analytic functions CW(GR)' or by the space of distri­butions C-oo(GIR):

2.6. Lemma. The inclusions CW(GR) '---+ COO(GR) '---+ C-oo(GR) induce topological isomorphisms

HOm(g,KR) (M', CW(GR)) ~ HOm(g,KR) (M', COO (GR))

HOm(g,KR) (M', C-oo(GIR))'

One can deduce this from the regularity theorems for elliptic V-modules by embedding MG( M) into Homva (Va ®U(g) M', Coo (GIR)) - in fact, the two spaces coincide if GR is connected - and noting that the Va-module Va ®U(g) M' is elliptic on GR , in the sense that its characteristic variety intersects the conormal bundle TOR G only along the zero section.

The lemma and the mere existence of globalizations imply that the ac­tion of GIR on MG(M) defines an admissible representation of finite length, which globalizes the Harish-Chandra module M. By construction, MG is a functor, and thus a right inverse to HC.

There is a dual construction, as follows. We now regard C8"(GIR ), the space of compactly supported, Coo functions, as right ({I, KIR)-module via right translation. As a right module, it can be tensored, simultaneously over U({I) and the group ring C(KIR1, with the left ({I, KJR)-module M. This tensor product is naturally a quotient of the tensor product over C, and therefore inherits a topology. We define

mg(M) = largest separated quotient of C8"(GR) ®(g,KR) M. (2.7)

We shall see later, as a consequence of our arguments, that the topology on the tensor product is Hausdorff, so the phrase "largest separated quotient" turns out to be unnecessary in the end. The left translation action of GR on C8"(GIR) induces a continuous, linear action on mg(M), and the resulting representation is another functorial globalization of the Harish-Chandra module M.

2.8. Theorem. The functors MG, MG are, respectively, left and right adjoint to HC. They are each other's dual, i.e., (mg(M))' ~ MG(M') and (MG(M))' ~ mg(M'). Both mg and MG are topologically exact functors.

462 M. Kashiwara and W. Schmid

The first assertion can be rephrased as follows. For V E R( GR) and M = HC(V), there exist functorial morphisms

mg(M)~ V, V L MG(M) such that

HC(o:) and HC({3) induce the identity on M = HC(V). (2.9)

Both 0: and (3 are injective, since the functor HC is faithful. Thus any globalization of a Harish-Chandra module M lies sandwiched in between mg(M) and MG(M), and that is the reason for calling the two functors the minimal globalization and maximal globalization. The duality between the two globalizations follows formally from (2.6) and the closed graph theorem of L. Schwartz: a surjective morphism in the category of locally convex, Hausdorff topological vector spaces, from a Suslin space to an inductive limit of Banach spaces, is necessarily open; moreover, CQ"(GJR) has the Sus lin property [Sz,T].

In slightly different, but equivalent form, Theorem 2.8 and the com­panion statements (2.12, 2.13) below were announced in [S2]. Except for (2.12) and topological exactness, they can be readily inferred from known results. Topological exactness and (2.12) are more delicate; the arguments outlined in [S2) deduce them from certain lower bounds on the matrix coef­ficients of Harish-Chandra modules. In this note, they will become natural consequences of our proofs of the conjectures (1.1b-f).

The minimal globalization functor first appears in the work of Litvinov­Zhelobenko and Prichepionok [Pl. Two other functorial globalizations were constructed by Casselman-Wallach [C2,W). Although their terminology differs, we shall refer to these two globalizations as the COC) and distribution globalization, since they relate to COC) functions and distributions in the same way the minimal and maximal globalizations relate to real analytic functions and hyperfunctions - cf. (2.12).

We choose a minimal parabolic subgroup 1\ of GR, and let P denote its complexification. The variety Y of G-conjugates of p = Lie(P) is a generalized flag variety, which contains the GR-orbit of p as real form:

YR = GR/1\ c...-. GIP = Y. (2.10)

Each irreducible, finite dimensional1\-module E associates an irreducible, GJR-equivariant, real analytic vector bundle E ~ YR to the principal bundle GIR ~ GR/1\ = YR. The KR-finite sections of E constitute a Harish­Chandra module,

(2.11)

Equivariant Derived Category and Representations 463

Collectively, Harish-Chandra modules of this type make up the principal series. Both CW (YIR , E), the space of real analytic sections, and C-w (YIR , E), the space of hyperfunction sections, have natural Hausdorff topologies - the latter because YR is compact - and continuous OR-actions. The resulting representations globalize the Harish-Chandra module (2.11). Hence, by (2.9), they contain its minimal globalization, and in turn are contained in its maximal globalization.

2.12. Theorem. The natural inclusions induce topological isomorphisms

mg (Ind~::~~nPa) (E») ~Cw (YR, E) ,

C-w(YR,E)~MG (Ind~::~~nPa)(E»)

This statement implies Helgason's conjecture; conversely, it was in­spired by the original proof of Helgason's conjecture [KMOT], and by Casselman-Wallach's construction of canonical globalization. In effect, Casselman-Wallach take the analogue of (2.12) as the point of departure of their definition: the spaces of Coo and distribution sections of E consti­tute, respectively, the Coo globalization and the distribution globalization of the Harish-Chandra module (2.11), but it then takes considerable ef­fort to show that these are concrete instances of functorial constructions [C2,W].

Now let V E R( OR) be a Banach representation. The space of analytic vectors VW c V has a natural inductive limit topology and a natural, continuous OR-action. If the Banach topology on V is reflexive, the dual representation is continuous also [HCl], so it makes sense to define the space of hyperfunction vectors, V-w =deJ «V,)w)'. This space contains V, and has a natural Frechet topology and continuous OR-action. Both VW and V-w globalize the Harish-Chandra module HC(V) , and thus lie between the minimal and the maximal globalization.

2.13. Theorem. The natural inclusion mg(HC(V» <-+ VW is a topo­logical isomorphism. Dually, if V is a reflexive Banach space, V-w <-+

MG(HC(V» is a topological isomorphism.

This formally implies (2.12), since CW(YR' E) and C-W(YR , E) can be identified with the spaces of, respectively, analytic and hyperfunction vec­tors for any of the reflexive Banach OR-modules LP(YR , E), 1 < p < 00.

Logically, (2.13) follows from (2.12), in conjunction with [W, §5.8] and the main theorem in [HS].

464 M. Kashiwara and W. Schmid

3. A derived category of representations

In this section we introduce a derived category of group representations. Much of the formalism, though in a different setting, is already known - cf. [BBD,L): while the category of topological vector spaces fails to be abelian, it is an exact category, and that suffices for our purposes. However, to keep the discussion short, we shall stay within the context of topological vector spaces, even though our arguments apply more generally in the setting of exact categories. We are indebted to J.-P. Schneiders for helpful conversations about the matters treated below.

For the moment, H will denote an arbitrary Hausdorff topological group. Eventually, OR will play the role of H, but H can also be the trivial group {e}, for example, in which case the derived category of repre­sentations reduces to the derived category of topological vector spaces.

Let TV be the category of locally convex, Hausdorff topological vec­tor spaces and continuous linear maps, and:F an additive full subcategory of TV. We consider the category :FH whose objects are vector spaces belonging to :F, together with a continuous, linear action of H; its mor­phisms are H-invariant, continuous linear maps. Next we form the category Cb (:F H) of bounded complexes in :F H, and the quotient category Kb (:F H), which has the same objects as Cb(:FH), but with homotopic morphisms identified. We call a complex (X,dx) in Cb(:FH) exact if, for each n, d~-l : xn-l -t Kerdx is an open, surjective map, relative to the subspace topology on Ker dx. 3.1. Lemma. Let I: X -t Y, g: Y -t X be morphisms in Cb(:FH)' and let M(f) denote the mapping cone 01 I. a) II go I is homotopic to idx, and il Y is exact, then so is X. b) II X and Y are exact, then M(f) is exact also.

Because of a), the notion of exactness descends from Cb(:F H) to Kb(:F H): if two complexes are isomorphic in Kb(:F H), then either both or neither are exact. Statement b) implies that N(:F H)' the full subcategory of Kb (:F H) consisting of exact complexes, is a null-system in the terminol­ogy of [KSa, §1.6). Thus one can "divide" Kb(:FH) by N(:FH)' We let Db(:F H) denote the resulting triangulated category; this is our (bounded) derived category of H-representations. We call a morphism I in Kb(:FH) a quasi-isomorphism if it has an exact mapping cone. In that case, I in­duces an isomorphism in the category Db(:FH)' A complex X in Kb(:FH) becomes zero in Db(:F H) precisely when it is quasi-isomorphic to zero, i.e., when it is exact.

The cohomology groups of an exact complex vanish. Consequently, the cohomology functors on Cb(:FH) determine functors on the level of Db(:FH);

Equivariant Derived Category and Representations 465

they take values in the category V of vector spaces without topology:

(3.2)

From now on, we assume that the category F is hereditary, in the following sense:

if V belongs to F, and if W c V is a closed subspace,

then Wand V/W also belong to F. (3.3)

If in addition the open mapping theorem holds in the category F - for example, if F is the category of Frechet spaces - then exactness of com­plexes reduces to the vanishing of cohomology, and a morphism in Db (F H) is an isomorphism if and only if it induces isomorphisms on the level of cohomology.

The hypothesis (3.3) allows us to introduce two pairs of truncation functors qr$n, qr?n and 8r $n, 8r ?n on the category Cb(F H), as follows:

qr$n(x)

qr?n(x)

sr$n(x) =

= X n- 2 X n- 1 K dn 0 ... - - - er x - - ... O K ..m-l X n- 1 Xn

... - - eruX - - - ... X n- 1 Xn -I dn 0 ... - - - m x- - ...

... _ 0 _ Xn/lm~-l _ xn+l _ X n+2 _ ... , (3.4)

where 1m d'X denotes the closure of 1m d'X, equipped with the subspace topology inherited from xn+l. Both are compatible with homotopy and quasi-isomorphism, and hence define truncation functors on Db(F H)' We set

q D$O(FH)

q D?O(F H)

S D$O(FH)

SD?O(FH)

{X E Db(FH) ; qr$OX - X is an isomorphism},

{X E Db(FH) ; X - qr?°X is an isomorphism},

{X E Db(FH) ; 8r $OX - X is an isomorphism},

= {X E Db(FH) ; X _ 8 r ?°X is an isomorphism}.

(3.5)

3.6. Theorem. Both (qD$O(FH), qD?O(FH)) and (8D$O(FH), sD?O(FH)) are t-structures on Db(FH)'

We let Q(FH) = qD$O(FH) n qD?O(FH), S(FH) = sD$O(FH) n 8 D?'O (F H) denote the hearts of these two t-structures. By construction, they are abelian categories. Each object of Q(F H) is isomorphic to the

466 M. Kashiwam and W. Schmid

cokernel of a morphism f in F H - viewed as morphism in Q(F H) - such that f is set-theoretically injective. This is our reason for calling the first t­structure the "quotient" structure. Dually, every object in the heart S(F H) of the "sub" t-structure is isomorphic to the the kernel of a morphism in F H

with dense image. One can show that the bounded derived categories built from Q(F H) or S(F H) are equivalent to Db(F H), but this is not crucial for our purposes.

The functors

(qr:50 0 qr?:o)(X[nJ), ,

(B r :50 0 Br?:O)(X[n])

take Db (F H) to the hearts of the two t-structures. We note:

On the other hand,

HO(qHn(X)) = Hn(x) ,

Hk(qHn(X)) = 0 if k =/: o.

HO(BHn(x)) = Kerdx/1mdX 1,

Hl(BHn(x)) = Imdx/Imdx ,

Hk(BHn(x)) = 0 if k =/: 0,1 .

For X E Db(F H) and nEZ, the following conditions are equivalent:

(3.7)

(3.8a)

(3.8b)

a) Imdx-1 is closed in X n , and dX-1 is an open map onto its image;

b) qHn(x) is isomorphic to an object in FH,

c) SHn-1(X) is isomorphic to an object in FH. (3.9)

The natural functor 'IjJ : F H --+ Db(:F H) is fully faithful. Hence, if the con­ditions (3.9) are satisfied, qHn(x) and SHn-1(X) are well defined objects in :F H. If these conditions hold for every n, we say that X is strict; in that case, qHn(x) = BHn(X) for all n.

Finally, let us suppose that H = Cit is a reductive Lie group. We shall say that X E Db(:FGR) has the property MG - respectively, the property mg - if X is strict and all its cohomology groups can be realized as maximal - respectively, minimal - globalizations of Harish-Chandra modules.

Equivariant Derived Category and Representations 467

4. Quasi-equivariant V-modules

We recall the definition of a quasi-equivariant V-module [K3, BB3, BBM]. Let G be an algebraic group with Lie algebra g, and X a smooth algebraic variety with an algebraic G-action - or algebraic G-manifold for short. We write J-l for the action morphism, p for the projection from G x X to X,

J-l:GxX-+X, J-l(g,x)=gx, p:GxX-+X, p(g,x)=x, (4.1)

and we consider the three maps

qj:GxGxX-+GxX, l~j~3,

ql(gl,g2,X) =(gl,g2X), q2(g1.g2,X) = (glg2,X), q3(g1.g2,X) = (g2,X).

(4.2)

Then J-l 0 ql = J-l 0 Q2, po Q2 = po Q3, and J-lO q3 = pO ql·

As usual, Vx will refer to the sheaf of linear, algebraic differential oper­ators on X. We let Oal8lVx denote the subalgebra Oaxx ®p-1ox p-1Vx of Vaxx. A quasi-G-equivariant Vx-module, by definition, is a Vx­module VJ1, together with the datum of an Oa 181 Vx-linear isomorphism {3 : J-l* VJ1~p* VJ1, such that the diagram

q;f3 q"2J-l* VJ1 q"2P* VJ1

1// II/

qiJ-l* VJ1 qrf3

Qip* VJ1 ~ qjJ-l* VJ1 q;f3

Qjp* VJ1 I

(4.3) commutes. If {3 is linear even over Vaxx, this reduces to the usual defini­tion of a G-equivariant Vx-module.

The definition of quasi-G-equivariance can be loosely paraphrased as follows. For 9 E G, let J-lg : X -+ X be translation by g. The datum of {3 consists of a family of isomorphisms of Vx-modules {3g : J-l; VJ1~VJ1, depending algebraically on g, and mutiplicative in the variable g.

As example, we mention the sheaf of sections O(E) of (E, VE), a G­equivariant algebraic vector bundle E with a G-equivariant, algebraic, fiat connection VE. Then Vx acts on O(E) via the fiat connection, and the resulting Vx-module is quasi-equivariant. The Lie algebra 9 of G acts on sections of the G-equivariant vector bundle E by infinitesimal translation; on the other hand, each A E 9 determines a vector field on X, again by infinitesimal translation, and as such also operates on sections via the connection. When these two actions of 9 coincide, the Vx-module O(E) is

468 M. Kashiwara and W. Schmid

G-equivariant, and one calls the sheaf of flat sections a G-equivariant local system.

If G acts transitively on X, equivariant vector bundles with flat, equivariant connection are the only examples of quasi-G-equivariant Vx-modules coherent over Ox: the category of quasi-G-equivariant, Ox-coherent Vx-modules coincides with the category of pairs (E, VTE), consisting of a G-equivariant algebraic vector bundle E with a G­equivariant, algebraic, flat connection VTE, and flat, G-equivariant bun­dle maps between them. The latter category, in turn, can be identified with the category of finite dimensional, algebraic (g, H)-modules, where H c G denotes the isotropy subgroup at some reference point Xo EX. The preceding statements remain correct even without the hypothesis of coherence, provided one allows for algebraic vector bundles of possibly infi­nite rank, and possibly infinite dimensional algebraic (g, H)-modules [K3J: if H eGis a closed subgroup, there exists a natural equivalence between ModG(VG/ H ), the category of quasi-coherent, quasi-G-equivariant V G/ H -

modules and G-equivariant morphisms, and Mod(g, H), the category of algebraic (g, H)-modules and (g, H)-invariant linear maps,

Here JeH denotes the ideal sheaf of the point eH; the isotropy group H op­erates on the "geometric fiber" rot /JeH rot via the equivariant Ox-module structure, and U(g) via an action which we shall now describe.

We return to the general case of a quasi-G-equivariant Vx-module rot. Again, the Lie algebra g acts on rot in two ways. First, via g -+ r( ex) <-+

r(vx ) and the Vx-module structure, and secondly, through differentiation of the G-action when we regard rot as a G-equivariant Ox-module. We denote the former action by av, the latter by at, and we set I = at - av. Then

"I : g ---+ Endvx (rot), and

I is a Lie algebra homomorphism. (4.5)

Thus I turns rot into a (Vx,U(g»-bimodule. The quasi-G-equivariant Vx-module rot is G-equivariant as Vx-module precisely when 1==0.

In our discussion of quasi-equivariance so far, no particular assumptions were imposed on the algebraic group G and the G-manifold X. From now on, however, we shall require G to be an affine algebraic group - which will be the case if G is reductive, as was assumed in the introduction -and X a quasi-projective variety. These hypotheses will enable us to define direct and inverse images, under G-equivariant morphisms, of objects in

Equivariant Derived Category and Representations 469

the derived category of quasi-G-equivariant Vx-modules. A crucial tool is the following result of Sumihiro ISu]:

4.6. Proposition. Under the hypotheses just stated,

a) X has a G-equivariant, open embedding into a projective G-manifold X;

b) the G-equivariant projective completion X admits an ample, G­equivariant line bundle.

As before, we let Moda(Vx) denote the category of quasi-coherent, quasi-G-equivariant Vx-modules and G-equivariant morphisms between them; ModG'h(Vx) shall denote the full subcategory ofVx-coherent mod­ules. The analogues of locally free Vx-modules in the quasi-equivariant setting are modules of the form Vx ®ox ij, for some coherent, locally free, G-equivariant Ox-module ij. We write Modg (Vx) for the full subcategory of Moda(Vx ) consisting of such locally free objects. As a consequence of Sumihiro's result, on can show:

4.1. Lemma. Every 9Jl E ModG'h(Vx ) has a finite left resolution by objects in Modg (Vx ).

Standard arguments in homological algebra imply that

every !m E Moda(Vx) can be embedded into an injective object. (4.8)

Injective objects in Moda(Vx ) are injective also as objects of ModG( Ox), the category of G-equivariant Ox-modules. In this latter category, direct image under a G-equivariant morphism of quasi-projective G-manifolds, as well as the global Ext functors behave as they do in the non-equivariant case.

We now consider the bounded derived categories D~(Vx), D~,coh(VX)

built from the abelian categories ModG(Vx) and ModG'h(Vx). Alterna­tively, the latter derived category can be described as the full subcategory of D~(Vx) consisting of complexes whose cohomology lies in ModG'h(Vx).

In view of the discussion in the preceding paragraphs, direct and inverse image functors can be defined on the bounded derived categories, as follows. Let f : X ---+ Y be a G-equivariant morphism between the quasi-projective G-manifolds X, Y. Then

defines a right exact functor from ModG(Vy ) to ModG(Vx ), which can be left derived in the bounded derived category:

(4.10)

470 M. K ashiwam and W. Schmid

The usual definition of direct image,

L rot ~ Rf.(Vy +-x ® rot),

Vx (4.11)

has meaning also in the G-equivariant case, if properly interpreted: Vy+-x has a deRham-type resolution by (I-IVy, Vx)-bimodules which are flat over Vx; if Vy+-x is replaced by this resolution, and if rot is represented by an injective complex, then (4.11) makes sense in the bounded derived category. We use the customary notation

(4.12)

for this V-module direct image functor. If f is smooth, Lj* sends D~oh(Vy) into D~oh(VX)' and if f is projec­

tive, If maps D~oh(VX) into D~oh(Vy). If f is both smooth and projective, the two functors are each other's left and right adjoints when restricted to the coherent subcategories, up to a shift in degree:

4.13. Theorem. Suppose f : X ----+ Y is both smooth tive, and let dimX/Y denote the dimension of the fibers. rot E D~,coh(VX) and IJ1 E D~,coh(Vy),

and projec­Then, for

a) HomDb(Vy)(IJ1, (rot) ~ HomDb(vx)(Lj*IJ1[-dimX/Y),rot) G 1f G

b) HomD~(Vx)(rot,Lj*IJ1) ~ HOmD~(VY)(i rot[dimX/Y),IJ1),

functorially in rot and IJ1 .

We shall apply these results about quasi-equivariant V-modules in the setting of the flag variety X of a reductive group G, and we shall need them also in the twisted case. It is possible to develop them systemati­cally for quasi-G-equivariant modules over a G-equivariant ring of twisted differential operators - for a general discussion of twisting, see [K3). On the flag variety, twisted sheaves are particularly easy to visualize; in effect, they are monodromic sheaves on the "enhanced flag variety" [BB3). The extension of the results above to the twisted case on the flag variety2 is therefore straightforward, and we shall use this more general case without further elaboration.

2 More precisely, on products of the flag variety with another variety, with the twisting confined to the flag variety factor.

Equivariant Derived Category and Representations 471

5. Equivariant derived category and GR-modules

The equivariant derived category was introduced by Bernstein-Lunts [BL); a concise summary can be found in [MUV1. We now recall the properties that are relevant for our purposes, but only in as much generality as is needed here.

Let X be a real or complex algebraic variety, considered as topological space in its Hausdorff topology, and G a linear, real or complex algebraic group acting on X. We write Mod(Cx) for the category of sheaves of C­vector spaces, Db(CX) for the bounded derived category Db(Mod(Cx »; this differs from the notation of the introduction, but is consistent with the conventions in the previous section. In the real algebraic case, D~_c(Cx) will denote the full subcategory of complexes with lR-algebraically con­structible cohomology; similarly, in the complex case, D~_c(Cx) is the bounded derived category of sheaves with C-algebraically constructible co­homology - cf. [KSal, where the analytic case is treated; the algebraic case is considerably simpler in most respects.

We choose a tower of connected real, respectively complex algebraic G-manifolds

(5.1)

such that

a) G acts freely on each Vn ;

b) for each p > 0, HP (Vn, q = 0 if n» 0 . (5.2)

For example, a sequence of Stiefel manifolds Vn will do, since the group G is assumed to be linear. Because of (5.2a), the quotients G\(X x Vn ) exist as algebraic varieties. We let Pn : X X Vn -t X denote the projection onto the first factor, qn : X X Vn -t G\(X x Vn) the quotient map, in: X X Vn '-+

X X Vn+1 the natural inclusion, and jn : G\(X x Vn) '-+ G\(X x Vn+d the map induced by in.

Objects of the bounded G-equivariant derived category Db(Cx) are, by definition, quadruples S = (Soo, (Sn), (4)n), ("pn», with Soo E Db(Cx), (Snh:5n<oo a sequence with Sn E Db(Cc\(XXVn», and (4)nh:5n<oo, ("pnh:5n<oo sequences of isomorphisms

,/,. .-1S ~ S 'l'n: I n n+1--t n, (5.3a)

such that, for each n,

472 M. K ashiwam and W. Schmid

'l/Jn : p;;lSoo~q;;lSn coincides with the composition of

-1 C' ·-1 -1 C' I ·-1 -1 S Pn Voo ~ ~n Pn+1v oo .-1.1,-1 ~n qn+1 n+1

~n 'f'n+1

(5.3b)

Morphisms in this category are pairs (1]exH (n)) , consisting of a single mor­phism "'00 and a sequence of morphisms (nh::;n<oo between correspond­ing objects, which are compatible with the consistency conditions (5.3a). The category Db(Cx) inherits the structure of triangulated category from Db(CX) and the Db(CG\(xxVn »'

The G-equivariant bounded derived category of lR-algebraically con­structible sheaves, Db,R_c(CX), is the full subcategory of Db(Cx) con­sisting of objects S whose components Soo, Sn belong to the appropriate derived categories of constructible sheaves. Analogously, in the complex algebraic case, there is a G-equivariant derived category of C-algebraically constructible sheaves, Db,c_c(Cx).

As in the previous section, we shall need to work also with twisted equivariant derived categories. A systematic discussion can be found in [K3]. In the applications we have in mind here, the twisting will take place only on flag varieties. This case is particularly simple, so we shall not go into further detail.

The direct and inverse image functors RI., RI" 1-1, I' correspond­ing to a G-equivariant, algebraic morphism I : X - Y exist in the G­equivariant case, and also in the G-equivariant constructible case. They have the same formal properties as in the usual bounded derived category Db(Cx ).

Now let H c G be a closed, algebraic subgroup. The tower {Vn } that was used to define the G-equivariant derived category can be used also for H, and thus, via inverse image from the G\(X x Vn ) to the H\(X x Vn ),

one obtains the restriction functor

(5.4)

If H is normal in G, and if H acts freely on X, Db(Cx) ~ Db/H(CH\X)' Applying this twice, viewing first G and then H as normal subgroup of G x H, and letting X x G play the role of X, results in equivalence of categories

G b( ~ b( ) IndH : DH Cx)->DG CXX(G/H) , (5.5)

Equivariant Derived Category and Representations 473

"induction from H to G"; here G acts diagonally on X x (G/H). The terminology is appropriate: Let p : X x (G / H) -+ X denote the projection; then Rp* 0 Ind~ and Rp! 0 Ind~ are, respectively, the right and almost left adjoint functors of Res~ - almost left adjoint in the sense that the adjointness relation involves a dimension shift and tensoring by the stalk of the orientation sheaf of G / H at the identity coset. Here, again, everything that was said remains valid in the twisted case.

Let us recall the definition of the Matsuki correspondence for sheaves. We adopt the notation of the introduction: X is the flag variety of the complex reductive group G, GR is a real form of Gan, KR C GR a maxi­mal compact subgroup. We fix a linear function A on the universal Cartan algebra, and let Dt-,>.(Cx), DbR,>.(Cx) denote the equivariant derived cat­egories with twist A. For equivariant derived categories on products of X with another space, the subscript >. shall refer to the category of sheaves with twist>. along the factor X, but without twist along the other factor. Since K and GR operate on X with finitely many orbits, constructibility for objects in Dt-,>. (Cx), DbR,>. (Cx ) comes down to finite dimensionality of the stalks of the cohomology sheaves. The inclusion

(5.6)

realizes the Riemannian symmetric space SR as real form in the affine symmetric space S. Further notation:

p:XxS-+X, q:XxS-+S (5.7)

are the projections onto the two factors. In [K2], it was conjectured that

<P: D~,>.,c_c(Cx) -+ DbR,>.,R-ACx ) ,

<p(T) = Rp! ((ResgRInd~(T») ® (q-1i*i!Cs)) [2dimS] (5.8)

defines an equivalence of categories. This was proved by Mirkovic-Uzawa­Vilonen3 [MUV]. Note that i!Cs ~ orsR/s[-codimRSR] ~ orsR[-dimS], since SIR lies as real form in the complex manifold S.

The key to our proof of (1.Ib-f) is the definition of R Hom~~ (9J1 ®

T, Ox) as an object in the derived category of Frechet GR-modules Db(:FGR)' for 9J1 E Db,coh('DX ) and T E DbR,IR_c(Cx ), For this pur­pose, we consider - slightly more generally than in the previous paragraph - an affine algebraic group G, a quasi-projective G-manifold X, and a real

3 [MUV] establish a slightly different equivalence, but their arguments can also be used to show that (5.8) is an equivalence.

474 M. K ashiwam and W. Schmid

form GR in Gan. Since our construction involves both the algebraic and analytic structure, we shall now make notational distinctions, for example, between the structure sheaves Ox of the algebraic variety X and Oxan of the complex manifold xan. We write F for the category of Fnkhet spaces and FCR for the category of Frechet GR-modules.

Let us work backwards to justify and motivate our definition. As be­fore, Pn shall denote the projection from X x Vn to X, with {Vn } as in (5.1). If we disregard the topology and GR-action for the moment,

RHomvxxvJLp~rot ®p~lT,O(XXVn)an)

~ RHomvArot ®T,OXan ) ® Rr(Cvn);

hence, and because of (5.2b),

RHomvx(rot®T, Oxan ) ~ I~RHomvxxvn (Lp~ rot®p~lT, O(xxvn)an). n

(5.9) This, in effect, allows us to replace X by X x Vn ; in other words, we may as well assume that G acts freely on X. But then T is isomorphic to the inverse image 7r- 1S of some S E D~_ACCR\X) under the quotient map 7r : X ~ GR \X. Like any object in the latter derived category, S can be represented by a complex whose terms are finite direct sums of sheaves j,Cu, where j is the inclusion of some open, semi-algebraic subset U c GR\X, Thus T can be similarly represented, but the open subsets in question are then inverse images of open subsets of GR \X, i.e., they are GR­invariant open subsets of X. We now replace Oxan by the Coo Dolbeault complex A(O,.), to which it is quasi-isomorphic, and rot by a bounded complex of locally free quasi-G-equivariant Vx-modules - cf. (4.7). Again neglecting the topology and GR-action,

RHomvx(Vx ®ox ;j®j,Cu,A(o,.» ~ RHomox(;j®j,Cu,A(O,.»

~ Rr (U; (;j*)an ®Oxan A(O,.» ~ r(U; (;j*)an ®Oxan A(O,.» ,

(5.10)

for every rot = Vx ®ox ;j E Mod~ (Vx ), with ;j E ModG'h(Ox) locally free, and every inclusion j : U ~ X of a GR.-invariant, open subset U of X.

The complex on the right in (5.10) has a natural Frechet topology -the Coo topology for differential forms - and continuous GR-action. That, in conjunction with the acyclicity asserted by (5.10), makes it possible to

Equivariant Derived Category and Representations

define the functor

Db,coh{VX ) x DbR,R-C{CX ) - Db{:FGR) ' (rot,T) .- RHom~~{rot ®T,Oxan).

475

(5.11)

Formally, this functor is a projective limit with respect to the tower (5.1), and an inductive limit with respect to the choice of a complex in Mod~ (V x) quasi-isomorphic to rot, and choices of particular representatives of T.

The functor (5.11) interchanges the roles of direct and inverse images on its two arguments: let I : X -+ Y be a G-equivariant morphism between algebraic, quasi-projective G-manifolds X, Yj then

5.12. Theorem. If I is projective, there exists an isomorphism

RHom~~{rot ®1;;n1T,OXan)

~ RHom~~ (i rot ® T, Oyan ) !-dimXjYj,

functorially in rot E Vb,coh{VX) and T E DbR,R-c{Cy). II I is smooth,

RHom~~{Lrrot ®T,Oxan)

~ RHom~~{rot ®R(fan),T,Oyan)l-2dimXjYj,

functorially in rot E Vb,coh{Vy) and T E DbR,R_c{CX).

We end this section with the statement of our main technical result, from which the other results will follow. We return to our earlier hypothe­ses, with X denoting the flag variety of g, the Lie algebra of the connected reductive group G. The groups KR and K have the same meaning as in the introduction, i : Sa ~ S is the inclusion (5.6), and p, q are the projections from X x S to the two factors, as in (5.7). We view X, S as algebraic G-manifolds, and Sa as real analytic manifold.

The twisted version of the covariant Riemann-Hilbert correspondence !K3j establishes an equivalence of categories

DRx : D'K -eq,coh (Vx,>.) ~ D'K,_>.,c_ACx ) ,

DRx{rot) = R 1iomvx.>.(Ox{)'), rot), (5.13)

between D'K_eq,coh{VX,>.), the bounded K-equivariant derived category of coherent Vx,>.-modules,4 and D'K,_>.,c_c(Cx), the bounded K-equivariant,

4 Coherent K-equivariant VX,>.-modules are necessarily holonomic, since K operates on X with finitely many orbits.

476 M. Kashiwara and W. Schmid

C-constructible derived category with twist -,\. A word about the twists: 001 is a complex of sheaves on X, i.e., sheaves without twist, over the ring of twisted differential operators V x ,>.., whereas Ox('\} is a twisted sheaf, with twist '\, of VX,>..-modules. Thus it makes sense to apply the functor R 'Hom over Vx to this pair, and the result will be an object in the derived category with twist opposite to that of Ox (,\), since R 'Hom is contravariant in the first variable. The Riemann-Hilbert correspondence is compatible with induction: if

Ind~ : D'K-eq,coh(VX,>..} ----. D~_eq,coh(VXXS,>..} (5.14)

is defined analogously to the induction functor (5.5), then

D'K -eq,coh (Vx,>..) Ind~ D~-eq,coh (Vx xs,>..)

DRx! !DRxxs (5.15)

D'K,_>.,c_c(Cx} Ind~ D~,_>.,c_c(Cxxs}

commutes. The following result is now essentially a consequence of Theo­rem 5.12, (5.15), and other functorial properties of the deRham functor.

5.16. Theorem. For 001 E D~,coh(VX,->..}, £ E D'K_eq,coh(VX,>..}, and C = DRx(£} E D'K,_>..,c_c(Cx },

RHom~; (1 (Lp* 001 ®Oxxs Ind~(,C» ® i*i'Cs , Osan)

~ RHom~P (001 ®cl>(C},Oxan(-'\})[dimXj x.->'

as objects in Db(:FGR)'

On the left in this identity, the (complexes of) sheaves Lp* 001, Ind~(£} are modules over the rings of twisted differential operators Vxxs,->' and Vxxs,>., respectively, so their tensor product over Oxxs becomes a module for the ring of (untwisted) differential operators VXxs via the "twisted comultiplication"

VXxs ----. VXxs,->.. ®oxxs Vxxs,>.. (5.17)

The direct image of this tensor product is simply G-equivariant "integration over the fibers" of a complex in D~ coh(VXXS}. On the right hand side in (5.16), 001 is an untwisted modul~ over the ring of twisted differential operators Vx,->', and the sheaf of Cx-modules cl>(C} has twist -'\, so their

Equivariant Derived Category and Representations 477

tensor product - this time over ex - becomes a Vx._~-module with twist -A, i.e., with the same twist as Oxan(-A).

We need to comment on our notational convention concerning twists. To keep the discussion in the introduction brief, we tacitly incorporated the shift by p (=one half of the sum of the positive roots), as is customary in representation theory. In the context of V-modules, this p-shift would affect the definition of inverse image. Thus, beginning with the present section, we normalize twists so that A = 0 corresponds to the untwisted case.

If one disregards both the topology and GR-action, one can re-interpret the left hand side of the identification (5.16) as

where Cs: denotes the sheaf of hyperfunctions on SR. Thus (5.16) amounts to a Poisson transform from GR-modules, geometrically realized on the symmetric space SR, to the same GR-modules, but now realized on the flag variety X.

Under suitable ellipticity hypotheses, hyperfunction solutions are nec­essarily smooth. Concretely, the space Homvs(IJl,Cs.), for any IJl E

ModG'h(Vs), has a natural Frechet topology and continuous GR-action, as a consequence of (4.7), for example. If IJl is elliptic along SR, in the sense that its characteristic variety Ch(lJl) intersects the conormal bundle Ts. S of SR only in the zero section, then this CCX> solution space coincides with the space of hyperfunction solutions, as topologized GR-module:

5.1S. Proposition.

as objects in Q(.rG.) , provided Ch(lJl) nTs.S c TsS,

6. Proof of the conjectures

We now have the machinery in hand to prove (l.Ib-f), as well as (2.12) and the exactness of the functors mg, MG. The crux of the matter is to identify both sides in (5.16) explicitly for particular choices of rot and .c. Throughout this section, the notation of (5.16) shall remain in force.

For any Borel subalgebra beg, we identify the quotient b/[b, bj with the universal Cart an algebra by specifying the set of weights of g/b as the set of positive roots. This differs from the convention in [BB1, K3j, but

478 M. Kashiwam and W. Schmid

has the advantage of making dominant weights correspond to positive line bundles. In particular, if p denotes one half of the sum of the positive roots, then

Ox(2p) ~ (n~)-1 (d=dimX) (6.1)

is the reciprocal of the canonical sheaf. The equivalence of categories (4.4) induces an equivalence of derived

categories

(6.2)

We note that the original definition of 4> can be re-interpreted as the V­module inverse image functor (4.9) corresponding to the inclusion {eK} '---+

G / K = S. Our next lemma follows from base change in the Cartesian square

x ----+ XxS

! ! {eK} ----+ S,

applied to the Vs-module Lp* rot ®ox Ind~(.C), and with VX,->.. ®ox (n~ )-1 in the role of rot.

6.3. Lemma. For rot = Vx,->"®Ox (n~)-1 and.c E D'k_eq,coh(VX,>..) ,

The characteristic variety of any ')1 E ModG'h(Vs) is G-invariant and intersects T:KS ~ (g/t) .. exactly in the characteristic variety5 of the finitely generated (g, K)-module 4>(')1). If, moreover, ')1 is annihilated by an ideal of finite codimension in Z(g) - equivalently, if 4>(')1) is a Barish-Chandra module [BCll - Ch( 4>(')1» lies in the nilpotent cone when one identifies gOo ~ 9 via a non-degenerate, Ad-invariant symmetric bilinear form. On the other hand, TSRS n T:KS ~ (gR/tR)* consists of semisimple elements, so the assumptions on ')1 ensure that ')1 satisfies the ellipticity hypothesis of (5.18). Recall the notation (2.3) for the dual of a Barish-Chandra module. At this point, the definition of the U(g)-module structure (4.5) and the definition of the functor MG imply:

5 i.e., the "associated variety" in the terminology of [Vol.

Equivariant Derived Category and Representations 479

6.4. Lemma. Suppose S)'t E Morl'Jh(Vs) is annihilated by an ideal of finite codimension in Z(g). Then cP(S)'t) is a Harish-Chandm module, and

as objects in Q(FcR ).

We now combine the previous two lemmas with theorem (5.16). We suppose that .e E D'K_eq,coh(VX ,>.) satisfies the vanishing condition

Hn(x,.e) = 0 forn:/=O, (6.5)

and set C = DRx(.e), as before. Then HO(X,.e) is a Harish-Chandra module, hence

6.6. Corollary. Under the hypotheses just stated,

MG(If(x, .en ~ q If (RHom!;~._JVx,->. ®ox (ni )-1 ® cJ>(C) , Oxan (-,x»)[dJ) ,

as objects in Q(FCR)'

Disregarding both the topology and GR-action for the moment, we can make the further identifications

RHomvx._" (Vx,->. ®ox (ni)-1 ® cJ>(C), Oxan(-,x»)[dJ ~ RHomox «ni)-1®cJ>(C), Oxan(-,x»[dJ

~ Rr(R1iom(cJ>(C), Oxan(-2p-,x»)[dJ)j (6.7)

cf. (6.1). Here we view cJ>(C) as object in D'K,_2P_>.,c_ACx ), as we may: 2p is an integral weight, and this implies the existence of a canonical iso­morphism

D'K,_2P_>',C_C(CX ) ~ D'K,_>.,c_ACx ). (6.8)

We shall apply (6.7) more specifically in the case of a VX,>.-module .e which corresponds to the Harish-Chandra module (2.11) via the Beilinson­Bernstein equivalence.

We use the notation of (2.10-2.11). Let 11' : X -+ Y denote the natural projection. Then D = 1I'"-1(YR) is the unique closed GIR-orbit in X. It is contained in the unique open K -orbit Q eX, and the Matsuki correspon­dence pairs the two orbits D, Q. The highest weight spaces E,..(x)/nxE,..(x), as x ranges over D, with b", = stabilizer of x in 9 and n", = [b"" b",], consti­tute the fibers of a GR-equivariant line bundle Lover D. This line bundle

480 M. Kashiwam and W. Schmid

extends to a K-equivariant algebraic line bundle over the K-orbit Q, and we refer to the extension by the same letter L. As is the case with all K-orbits in X, the inclusion of the open orbit i : Q ~ X is an affine mor­phism, so the 'V-module direct image of OQ(L) coincides with the sheaf direct image

(6.9a)

and the higher (sheaf) direct images vanish; here>. denotes the highest weight of E. Moreover,

(6.9b)

Proofs of these assertions can be found in [HMSW], for example. The Riemann-Hilbert correspondence relates the V-module direct im­

age to the direct image in the derived category of sheaves of (:-vector spaces, so

(6.lDa)

is the direct image of a K-equivariant, twisted local system on Q. The proof of (l.la) in [MUV] gives a description of (>(.c) for this particular sheaf .c, namely

(6.lDb)

with j denoting the inclusion D ~ X. We now use the fact that 1f : D--+ YlR is a real analytic fibration, with smooth complex projective fibers, and Bott's description of the cohomology of Oxon (-2p - >.) along the fibers, to conclude

Hn(Rr(R 1-l om«(> (.c) , Oxon (-2p - >')}[dJ))

"'" { C-W(YR, E* ® "maxr*y) if n = 0 - 0 if n =1= 0,

(6.11)

still without regard to the the topology and GR-action. Formally, the identity (6.11) should involve also the orientation sheaf of YlR, but the con­nectivity assumption about G implies that YR is orientable. In the derived category of Frechet GR-modules, vanishing of the ordinary cohomology forces exactness. Hence, in view of (6.6-6.8), we have shown:

Equivariant Derived Category and Representations 481

6.12. Proposition. If.c satisfies the hypotheses (6.9),

qII" (RHom~~._J'Dx,-A ®ox (n~ )-1 ® ~(.c), Oxan(-A))[dJ)

::::: {MG (Ind~::~;nPa)(E* ® I\maX(gJP)*)) ifn = 0 o if n =1= 0 .

In the next statement, Coo(GR)KR-Jini shall denote the space of Coo functions on Gilt which are KR-finite under the right action. We regard this space as a left (g, K)-module by composing the right action with the canonical anti-automorphism of g.

6.13. Theorem. For every Hansh-Chandra module M and every n =1= 0,

The isomorphism between the two Ext groups is formal. For any mem­ber M of the principal series, the vanishing of the higher Ext groups follows from (5.16) and (6.12). It suffices to prove the vanishing for irreducible Harish-Chandra modules. That can be done by downward induction on n - irreducible Harish-Chandra modules can be realized as submodules of modules belonging to the principal series [BB2, C1l; for large n, vanishing follows from the finiteness of the global dimension of 'Ds, or alternatively, from the analogous finiteness statement in the category Mod(g, K).

Theorem 6.13 implies the exactness of the functor MG, and by duality, also of MG. Indeed, we may replace Coo(GR) by Coo(GR)KR-Jini in the definition (2.5), and (g, KR)-invariance by (g, K)-invariance - the image of a KR-finite vector under a KR-invariant linear map is necessarily KR­finite also - so the theorem applies directly. The vanishing of the higher Ext groups means, in particular, that RHom(g,K)(M, Coo(GR)KR-Jini) E Db(FGR) is strict. That, in turn, insures that the induced topology on Cff (GIR) ®(9,KR) M is Hausdorff, so the phrase "largest separated quotient" in the definition (2.7) becomes unnecessary.

The isomorphisms (6.7) justify the following definition. For S E

DbR,A,IR-c(Cx), we regard RHom(S,Oxan(A)) as object in the derived category of Frechet GR-modules via the identification

RHom(S,Oxan(A)) ::::: RHom~P ('D'Dx>.®ox(n~)-1®S, Oxan(A+2p)). x.>' • (6.14)

According to (5.16), (6.6) and (6.13), this complex has the prop-

482 M. Kashiwara and W. Schmid

erty MG and satisfies (1.1b,d,f),6 at least if e = (DRx )-lep-l(S) E D'K_eq,coh(VX,->.-2P) has non-zero cohomology in only one degree; the general case follows by means of standard techniques in representation the­ory [SWj. The definition ofRr(S®Oxan( -'\-2p» as object in the derived category of DNF GR-modules is analogous and formally dual to the preced­ing case. The duality between the two globalization functors then implies the property mg and the assertions (lc,d,e) for the latter complex.

Until now, we have used (6.7) and (6.11) only to establish the vanishing of the higher Ext groups in (6.12). A careful examination of this chain of isomorphisms shows that the topology and GR-actions are preserved, Le.,

qHQ (RHom~~,_.\ (Vx,->. ®ox (n~-)-l ® ep(e), Oxan( -.\))[dj)

~ C-W(YR,E*®l\maxT*Y), (6.15)

as objects in Q(Fca), if.c is chosen as in (6.9). That, in conjunction with (6.6), implies the second half of Theorem 2.12 - the other half follows by duality.

7. Invariant systems of differential equations

In this section, Z shall denote a quasi-projective G-manifold, ZIR a GR­invariant real form, i : ZR "--+ Z the inclusion map, and !.lJt a coherent, quasi-G-equivariant Vz-module. We regard R Homvz (!.lJt, Cz:) as an ob­ject in the derived category of Frechet GR-modules Db(FcR) by making the identification

We shall show, under appropriate hypotheses, that this object has the property MG. In particular, the space of hyperfunction solutions of the restricted system on z.it will then have a natural Frechet topology and continuous GIR-action, and the resulting representation will be admissible, of finite length.

Recall the definition of the homomorphism I : U(g) -+ Endvz (!.lJt) in Section 4, and let Il-z : T* Z -+ g* denote the moment map. A calculation with a good filtration of !.lJt by G-equivariant coherent Oz-modules shows:

Ch(!.lJt) C Il-z 1 (Ch(U(g)jAnny(!.lJt))) , (7.2)

6 the reasons for the appearance of 2p in the present discussion and for its absence in the introduction were explained in Section 5.

Equivariant Derived Category and Representations 483

where Ann..y(001) denotes the annihilator of 001 in U(g) with respect to the action "{. If 001 is not only quasi-G-equivariant as 'Dz-module, but G-equivariant, then the entire augmentation ideal annihilates 001, so

The nilpotent cone N· c g. - i.e., the image of the nilpotent cone NEg when g* is identified with 9 by means of an Ad-invariant, nondegenerate symmetric bilinear form - is the variety defined by the augmentation ideal in 8(g)G, hence

Ch( 001) C Jlzl (N*) if 001 is Z(g) - finite j (7.4)

here Z(g)-finiteness means that some ideal of finite codimension I C Z(g) (= center of U(g» annihilates 001.

Borel subalgebras are solvable. Hence, by arguments in either [KMF] or [G, appendix],

Ch(001) nJlzl(b.l.) C T*Z is an involutive subvariety, (7.5)

for any Borel subalgebra b of g. We shall call 001 admissible if this involutive subvariety is Lagrangian, for every b - or equivalently, for some b, since the action of G preserves Ch( 001 ).

7.6. Theorem. Let 001 E Modah(Vz) be admissible and Z(g)-jinite, and S an object in the bounded equivariant derived category D~R,R_c(CZ)' Then RHom~~( 001 ® S, Ozan), as object in Db(:FGR)' has the property MG.

This statement neither involves, nor depends on, the existence of a GIR-invariant real form ZR. However, when such a real form does exist, the theorem, with i*i'Cz in place of S, provides the criterion alluded to atthe beginning of this section.

We begin the sketch of the proof of (7.6) with some general remarks. The forgetful functor

(7.7a)

has a left adjoint,

EQ : ModG('Dz) ---t ModG_eq(Vz) , (7.7b)

given by EQ( 001) = 001 /"I(g) 001 j here "{(g) 001 denotes the image in 001 of the quasi-G-equivariant 'Dz-module 9 ® 001. The functor EQ is visibly right exact.

484 M. Kashiwara and W. Schmid

Coherent, Z(g)-finite, quasi-G-equivariant Vz-modules admit finite fil­trations such that the successive quotients are modules with an infinitesimal character, i.e., modules on which Z(g) acts by a character. Also, if any two objects in a distinguished triangle in Db(:FG.) have the property MG, then so does the third. This allows us to assume, without loss of gener­ality, that rot itself has an infinitesimal character. To be consistent with our notational choices in Sections 5 and 6, we index characters of Z(g) by linear functionals A on the universal Cartan without the customary shift by p (= half sum of the positive roots); in other words, XA : Z(g) -t C denotes the character by which Z(g) acts on the Verma module with high­est weight A. Then XA = X,.. if and only if A + p is conjugate to J.L + p under the action of the Weyl group W. We let Mod&(Vz) denote the full subcategory ofModG(Vz) consisting of modules with infinitesimal char­acter XA' and Mod~h'A(Vz) the full subcategory of coherent modules in Mod&(Vz ). Because of our earlier assumption, rot belongs to one of these subcategories:

(7.8)

Replacing A + P by an appropriate W-translate, we can arrange

(0, A + p) ~ Z<o, for every positive coroot 0; (7.9)

we shall refer to this condition by saying that A + p is integrally dominant. For the moment, we do not assume (7.9). We shall consider modules

over the ring of twisted differential operators VXXZ,A on the product of the flag variety X with Z; the twisting is confined to the factor X, and is indexed by the parameter A. We write p : X x Z -t X, q : X x Z -t Z for the two projections. With these ingredients, we define functors

6. : Mod&(Vz ) ---t ModG-eq(VXxZ,A) '

\11 : ModG-eq{VXxZ,A) ---t Mod&{Vz) ,

6.(91) = EQ(VX,A ~ 91) ,

1JI{.c) = q.(.c). (7.10)

The fact that IJI is well defined requires verification: the G-equivariant projection q has projective fibers, so the sheaf direct image q. exists as a left exact functor between the categories of quasi-G-equivariant, quasi­coherent V-modules; a small calculation shows that the Vxxz,A-module structure of.c imposes the infinitesimal character XA on q.(.c).

By construction, 6. is right exact, IJI left exact. We note also that \11 is the right adjoint of 6.. The next statement, we shall see, formally contains the Beilinson-Bernstein equivalence of categories [BBI].

7.11. Theorem. If A + p is integrally dominant,

Equivariant Derived Category and Representations 485

a) Rnq.(,C) = 0, for every ,C E Mod(Vxxz,,x) and every n > 0; b) \}!6('Jl) ~ 'Jl, for every 'Jl E Mod&(Vz ).

If >. + p is both integrally dominant and regular, the functors 6, \}! define equivalences of categories, and are quasi-inverses to each other.

The proof of this theorem amounts to a reduction to the analogous statements in [BB1]. The vanishing ofthe higher derived images Rnq.(,C), for example, is local with respect to Z, so one may as well suppose that Z is affine. But then Rnq.(,C) is determined by its space of global sections, and that space coincides with the n-th cohomology of the quasi-coherent - for affine Z - Vx,,x-module P.('c), so the Beilinson-Bernstein vanishing theorem applies. The other assertions can be verified by similar arguments.

To make the formal connection with [BB1], let us look at an algebraic subgroup H c G. The theorem, with Z = G/H and>' + p integrally dominant regular, asserts an equivalence of categories

(7.12a)

But ModG-eq(VXxG/H,,x) ~ ModH-eq(VX,,x) for formal reasons - "induc­tion from H to G." On the other hand, the equivalence of categories (4.4) identifies ModG-eq(VXxG/H,,x) with Mod,x(g, H), the category of algebraic (g, H)-modules with infinitesimal character X,x. Thus (7.12a) is tantamount to the equivalence of categories

(7.12b)

This, of course, follows from the Beilinson-Bernstein equivalence and, in fact, reduces to it precisely when H = {e}.

Our next statement can be verified directly, by keeping track of the effect of the functors 6, \}! on characteristic varieties; both (7.3) and (7.5) are crucial ingredients of the argument.

7.13. Lemma. The functor 6 assigns a holonomic module in ModG-eq(VXxZ,,x) to any admissible module'Jl E Mod&(Vz ). Conversely, \}! assigns admissible modules to holonomic modules.

For the proof of (7.6), we may suppose that the admissible, quasi-G­equivariant Vz-module 9J1 satisfies the additional hypotheses (7.8-7.9). Because of (7.11) and (7.13), there exists a holonomic, hence coherent module ,C E ModG-eq(VXxZ,,x) , such that

486 M. K ashiwara and W. Schmid

Thus, as a consequence of the definition of the direct image functor (4.12), we obtain the isomorphism

(7.14)

in the derived category D~ coh('DZ ); here d, it should be recalled, de­notes the dimension of X. ' We now argue as we did in Section 5: for S E D~R,R_c(CZ)' and with £, = DRxxz(£),

RHom~~ (i (Lp*('Dx,_>. ®ox (01)-1) ®oxxz £) ®S, Ozan) [2dimZ]

~ RHom~~,_...('Dx,_>. ®ox (01 )-1 ® Rp*(£' ® q-1s) , Oxan( -A))[dJ,

(7.15)

as objects in Db(:FGR)' Except for the concrete choices of S and A, the object on the right in (7.15) coincides with the object (6.14), and thus has the property MG.

References

[BB1] A. Beilinson and J. Bernstein, Localisation de g-modules, C. R. Acad. Sci. Paris 292 (1981), 15-18.

[BB2] A. Beilinson and J. Bernstein, A generalization of Casselman's sub­module theorem in: Representation Theory of Reductive Groups, Progress in Mathematics, vol. 40, Birkhiiuser, Boston 1983, 35-52.

[BB3] A. Beilinson and J. Bernstein, A proof of Jantzen's conjecture, Advances in Soviet Math. 16 (1993), 1-50.

[BBD] A. Beilinson, J. Bernstein, and D. Deligne, Faisceaux pervers, Asterisque 100 (1982),5-171.

[BL] J. Bernstein and V. Lunts, Equivariant sheaves and functors, Lec­ture Notes in Math. 1578 (1994), Springer.

[BBM] W. Borho, J.-L.Brylinski, and R. MacPherson, Nilpotent Orbits, Primitive Ideals, and Characteristic Classes, Progress in Mathe­matics, vol. 78, Birkhiiuser, Boston, 1989.

[C1] W. Casselman, Jacquet modules for real reductive groups, in: Proc. of the International Congress of Mathematicians, Helsinki, 1980, 557-563.

[C2] W. Casselman, Canonical extensions of Harish-Chandra modules to representations of G, Can. Jour. Math. 41 (1989), 385-438.

[G] V. Ginzburg, g-modules, Springer's representations and bivariant Chern classes, Advances in Math. 61 (1986), 1-48.

Equivariant Derived Category and Representations 487

[GW] R. Goodman and N. Wallach, Whittaker vectors and conical vec­tors, Jour. F'unct. Anal. 39 (1980), 199-279.

[HC1] Harish-Chandra, Representations of semisimple Lie groups I, Trans. Amer. Math. Soc. 75 (1953), 185-243.

[HC2] Harish-Chandra, The characters of semisimple Lie groups, Trans. Amer. Math. Soc. 83 (1956), 98-163.

[HC3] Harish-Chandra, Discrete series for semisimple Lie groups I, Acta Math. 113 (1965), 241-318.

[He] S. Helgason, A duality for symmetric spaces with applications to group representations I, Advances in Math. 5 (1970), 1-154; A duality for symmetric spaces with applications to group represen­tations II, Advances in Math. 22 (1976), 187-219.

[HMSW] H. Hecht, D. Milici<~, W. Schmid, and J. Wolf, Localization and standard modules for real semisimple Lie groups I: The duality theorem, Invent. Math. 90 (1987), 297-332.

[HS] H. Hecht and W. Schmid, On the asymptotics of Harish-Chandra modules, Jour. Reine und Angewandte Math. 343 (1983), 169-183.

[HT] H. Hecht and J. Taylor, Analytic localization of group representa­tions, Advances in Math. 79 (1990), 139-212.

[K1] M. Kashiwara Character, character cycle, fixed point theorem, and group representations, in: Advanced Studies in Pure Math. 14 (1988), 369-378.

[K2] M. Kashiwara, Open problems in group representation theory, Pro­ceedings of Taniguchi symposium held in 1986, RIMS preprint 569, Kyoto University, 1987.

[K3] M. Kashiwara, Representation theory and V-modules on flag man­ifolds, A sterisque 173-174 (1989), 55-109.

[K4] M. Kashiwara, V-modules and representation theory of Lie groups, RIMS preprint 940, Kyoto University, 1993, to appear in Ann. de l'Institut Fourier.

[KMF] M. Kashiwara and T. Monteiro-Fernandes, Involutivite des varil~tes microcaracteristiques, Bull. Soc. Math. Prance 114 (1986), 393-402.

[KMOT] M. Kashiwara, A. Kowata, K. Minemura, K. Okamoto, T. Oshima, and M. Tanaka, Eigenfunctions of invariant differential operators on a symmetric space, Annals of Math. 107 (1978), 1-39.

[KSa] M. Kashiwara and P. Schapira, Sheaves on Manifolds, Springer, 1990.

[L] G. Laumon, Sur la categorie derivee des V-modules filtres, in: Alge­braic Geometry, Proceedings, 1982, Lecture Notes in Mathematics 1016 (1983), Springer, 151-237.

[Ma] T. Matsuki, Orbits on affine symmetric spaces under the action of parabolic subgroups, Hiroshima Math. J. 12 (1982), 307-320.

488 M. Kashiwam and W. Schmid

[Mo] H. Matumoto, Whittaker vectors and the Goodman-Wallach oper­ators, Acta Math. 161 (1988), 183-241.

[MUV] I. Mirkovic, T. Uzawa, and K. Vilonen, Matsuki correspondence for sheaves, Inventiones Math. 109 (1992), 231-245.

[P] S. J. Prichepionok, A natural topology for linear representations of semisimple Lie algebras, Soviet Math. Doklady 17 (1976), 1564-1566.

[SI]

[S2]

[Su]

[SVl]

[SV2]

[SW]

[Sz]

[T]

[Vol

[W]

W. Schmid, Homogeneous complex manifolds and representations of semisimple Lie groups, thesis, Berkeley 1967. Reprinted in: Representation Theory and Harmonic Analysis on Semisimple Lie Groups, Mathematical Surveys and Monogmphs 31 (1989), Amer. Math. Soc., 223-286. W. Schmid, Boundary value problems for group invariant differen­tial equations, in: Elie Cartan et les MatMmatiques d'Aujourd'hui, Asterisque numero bors series (1985), 311-322. H. Sumihiro, Equivariant completion, Jour. Math. Kyoto Univ. 14 (1974), 1-28. W. Schmid and K. Vilonen, Character, fixed points and Osborne's conjecture, Contemp. Math. 145 (1993), 287-303. W. Schmid and K. Vilonen, Characters, characteristic cycles, and nilpotent orbits, in: Geometry, Topology, and Physics, Interna­tional Press, Boston, 1994. W. Schmid and J. Wolf, Geometric quantization and derived func­tor modules for semisimple Lie groups, Jour. Funct. Anal. 90 (1990), 48-112. L. Schwartz, Sur Ie thooreme du graphe ferme, Compt. Rend. Acad. Sci. 263 (1966), 602-605. F. Treves, Topological Vector Spaces, Distributions and Kernels, Academic Press, 1967. D.Vogan, Gelfand-Kirillov dimension for Harish-Chandra modules, Inventiones Math. 48 (1978), 75-98. N.R. Wallach, Asymptotic expansion of generalized matrix entries of real reductive groups, in: Lie Group Representations I, Lecture Notes in Mathematics 1024 (1983), Springer, 287-369.

M. Kashiwara, Research Institute for Mathematical Sciences, Kyoto University, Kyoto 606, Japan.

W. Schmid, Department of Mathematics, Harvard University, Cambridge, MA 02138, USA.

Received April 12, 1994

Meromorphic Monoidal Structures

D. Kazhdan

In this article we restate some results of [KL] in a form which can be imme­diately generalized to the quantum case. The corresponding "quantization" is the joint work with Y. Soibelman. Unfortunately we have obtained a "quantization" of only the most elementary part of [KL]. We will use freely notations from [KL].

This reformulation is heavily influenced by the attempts to understand the remarkable paper [FR] of I. Frenkel and N. Reshetikhin.

Let g be a simple Lie algebra over C, h be the dual Coxeter number of g and ( , ) : g x g -+ C be the g-invariant bilinear form defined as (2h)-1 times the Killing form.

We shall assume, as we may, that g is given in terms of the generators ei,/i,hi , (i E I) and the Serre relations [ei,h] = 8ijhi j [hi,ej] = aijej, [hi, iiI = -aijlij [hi, hj ] = OJ (adei)-a;j+l(ej) = (adM-a;j+1(Ii) = O. Here (aij)i,jEI is the Cartan matrix. The form ( , ) satisfies (hi, hj) = aij and (ei' Ii) = 8ij .

Let g- be the Lie subalgebra ofg generated by the elements Ii, (i E I). Let ZI (resp. NI) be the set of all maps 1-+ Z (resp. 1-+ N). Let (bij )

be the matrix inverse to (aij)j for a, bE ZI, we set (a, b) = Lid bija(i)b(j).

For any a E NI, let Va = U(g-)/ (Li U(g_)I:(i)+l). Let Ya be the

image of 1 E U(g-) in Va. There is a unique g-module structure on Va such that g- acts by left multiplication, eiYa = 0 and hi(Ya) = a(i)Ya for all i E I. This g-module is simple and finite dimensional.

The g-module dual to Va is isomorphic to V=i where a -+ a is an involution of NI. There is a well defined involution i t-+ 1: of I such that a( i) = a(1:) for all i E I.

Let € be an indeterminate, and CD. be the free C-module of rank one with basis element D..

For 11, h E q€, e l ] we define {ft, h} E C to be the residue at € = o of the differential form hd(ft). We have {ft, h} + {h, It} = 0 and {fth,h} + {hh,Jt} + {hft,h} = 0 for all ft,h,h E q€,e l ]. Hence

490 D. Kazhdan

is a Lie algebra with bracket

[Je,!,e'] = I!'[e,e'] + {j,!'He,e')nj [Ie, n] = OJ [n, n] = 0

for I, I' E C[t, t- l ] and e, e' E g. The bracket can be also described by the formula

for any n, m E Z and any e, e' E g, together with the requirement that n is in the centre.

g is called the affine Lie algebra. Let g' be the Lie subalgebra C[t] ® g EB en of g and g+ c g' be the

Lie subalgebra C[t] ® g. We have a natural embedding e --+ e ® 1 of g into g and we will often

consider g as a Lie subalgebra of g. We denote by w : g --+ g the Lie algebra involution defined by w(tne) =

(-t)-ne,w(n) = -no Let U = U(g) be the universal envelopping algebra of g, and U ~f U(g),

U+ ~f U(g+) C U' ~f U(g') cUbe its subalgebras. Let () : U+ --+ U be the algebra homomorphism such that ()(n) = 0 and

()(f ® c) = I(O)e, I E C[t], e E g. Let I be the kernel of () and 1- ~f w(I).

For any integer n ~ 1 we define u;t ~f In C U, U;; ~f w(u;t) C U. For any Z E e* we denote by 1/1z : g --+ g the evaluation map defined

by the rules 1/1z(n) = 0, 1/1z(f ® c) = I(z)e, IE C[t, ell, e E g. It is clear that 1/1z is a Lie algebra morphism. We denote by Va,z the representation of g which is the composition of 1/1z and Va.

For any it = (al," " an), Z = (Zl,'" ,zn), ak E N1 , Zk E e*, Zk =1= ZI." for k =1= k', 1 ~ k, k' ~ n we define the representation Va,z of g as the tensor

product Va,z ~f ®~=l Vdk,Zk'

Lemma 1. a) For any it = (all"', an), Z = (Zl, ... , zn), Zk =1= Zk' 101' k =1= k' the representation Va,z is irreducible.

b) Any irreducible finite-dimensional representation 01 g is equivalent to Va,z 101' some it = (all" ., ak) E (NI)n, Z = (Zl," . , zn) E (c*)n, Zk =1= Zk' for k =1= k'.

c) The category C of finite dimensional representations of g is semisimple.

Given a U-module V and an integer n ~ 1 we define a subspace V(n) of V

V(n) = {v E Vlav = 0 'Va E u;t and dimcUv < oo}.

Meromorphic Monoidal Structures

We define V(oo) c V as the union of all V(n),n ~ 1.

Lemma 2. V (00) is a U -submodule of V.

We say that a U-module V is smooth if V = V(oo).

491

Let V be the category of U -modules, V+ the full subcategory of smooth modules, V ... the full subcategory of V of modules V such that 1 acts as ('" - h)Idv and v;t = V+ nv ....

We say that a finite dimensional U+ -module N is a nil- module if there exists n E N such that the U+.N = O. For any nil-module N and a number '" E C we extend the action of U+ on N to an action of U' on N in such a way that 1 acts as ('" - h)IdN and define

The map n --+ 1 ® n defines an imbedding N '-+ N'" and we will always consider N as a submodule of N"'. We say that N'" is a generalized Weyl module. Clearly N'" lies in v;t for all '" E C.

We apply the previous definition in the case when N = Va for some a E N1 regarded as U+ -module with ui acting as zero. In this case the U-module N'" is denoted by V~; it is called a Weyl module.

Lemma 3. a) The U -module V~ has unique irreducible quotient L~.

b) If '" ~ Q, then V~ is irreducible for all a E N1 .

c) If '" ~ Q20, then the U -module V~ has finite length and all its com­position factors are either isomorphic to L~ or are of the form L~, where a' E N1 is such that (a' + 1, a' + I) < (a + 1, a + I). Moreover L~ appears exactly once in a composition series of V~. In particular, V~ is irreducible for all '" ~ Q20.

In the remainder of this section we assume that '" ~ Q20.

Theorem 1. Let V be an object in V.... The following conditions are equivalent :

a) There exists a finite composition series of V with subquotients of the form L~ for various a E N1 .

b) V is a quotient of a generalized Weyl module.

c) There exists N ~ 1 such that dim V(N) < 00 and V(N) generates V as aU-module.

d) V is smooth and dim V(l) < 00.

We denote by OK the full subcategory of v;t of modules satisfying the conditions of Theorem 1.

Lemma 4. If '" ~ Q then any object in 0 ... is a direct sum of Weyl modules.

492 D. Kazhdan

For any U-module (p, M) we denote by M- the representation p of U on the same space M such that p(x) = p(w(x».

We denote by V- the category of U-modules M such that M- lies in V+ and by 0; c V- the category of U-modules M such that M- lies in

01<' For any nil-module N of U+ and /'i, E C we define NI< ~f (NI<)-. As

before, we have a natural imbedding N- '--+ NI< of U-modules where we identify N- with N as a vector space and (x, n) -+ w(x)n for all x E U, n E N.

For any g-module R we denote by Ro c R the subspace spanned by elements of the form ar, where a E g, r E R. We call the quotient space RA = R/ Ro the space of coinvariants of R. Given two g-modules M, N we denote by {M, N}g = M®N/(M®N)o the space of coinvariants of M®N and denote by IIM,N (or simply II) the natural projection II : M ® N -+

{M,N}g.

Let M,N be nil-modules /'i, E C and M ® N- ~ MI< ® NI< be the natural imbedding. Then we have a linear map II 0 i : (M ® N-) --+

{MI<,NI<}g, which factors through the map:Z: {M,N-}g --+ (MI<,NI<}g,

where {M,N-} ~f Homg(M ®N-,C).

Lemma 5. The map :z is an isomorphism.

For any M in VI< and n E N we denote by M(n) eM the subspace gen­

erated by vectors of the form xm, x E U+' m EM, define (M}n ~f M/M(n) and denote by 7I"n : M -+ Mn the natural projection.

Lemma 6. IfNI< is a generalized Weyl module, P is a finite-dimensional representation ofg and M = NI<®P then the natural morphismN®P--+ {M}t is an isomorphism.

Let £1< C VI< be the full subcategory of modules M such that dim (Mh < 00.

Lemma 7. a) For any M in £1< and any n E N we have dim (M}n < 00.

b) For any N in 01< and any P in C the tensor product N ® P lies in £1<' -def ~ -

For any M in £1< we define M = lim{M}n and denote by M c M the <-

subspace of smooth vectors. In other words, M = M(oo).

Theorem 2. For any M in £1< the smooth g-module M lies in 01<'

For any finite-dimensional g-module P and any g-module M in 01< we know that M ® P lies in £1<' Therefore we can define functors ®P from (')1< to itself

Meromorphic Monoidal Structures 493

Theorem 3. Assume that either", is an irrational number or that '" is mtional and its nominator is big enough (see [KL} §32). Then the functor @P is exact.

In the case when P = Vii,i', ii = {ak},z = {Zk}, ak E NI, Zk E C*, 1 :-::; k :-::; n, Zk =I Zk' for k =I k' we can identify canonically the functor @Vii,i' from OK to itself with the functor M - 'Y(M@V~l @ ... @V~J (see [KL] §4.2) in the case when C is connected and 'Y = bo, 'YZl> .. . , 'YZn' 'Y<Xl)

where 'Yo = I d, 'Y2 = (~ ~) and 'Y<Xl = (~ ~1). In particular, in

the case when Z = {Zk} where Zk, 1 :-::; k :-::; n are real numbers such that 0< ZI < '" < Zn < 00 we can identify canonically the g- module M@Va,z with M@Va1@",@Van (see [KL] §13.4) .

Therefore Theorem 3 follows immediately from the rigidity of the cat­egory OK (see [KL] §32).

Since the proof of the rigidity of the category OK is very involved, it will be interesting to find a more direct proof of the exactness of the functor @P for P in C. It is not difficult to check that the following conjecture implies directly the validity of Theorem 3.

Conjecture. For any M in OK. '" ¢. Q~o there exists an integer n 2:: 1 such that the restriction of M to U;; is free .

Let C, F be two strict monoidal categories (see [M]) and a be a functor from C to F. A monoidal structure P on a is by the definition a functorial isomorphism

PP,Q : a(P) ®:F a(Q) ~ o:(P ®c Q)

such that the diagram

lid®PQ,P

a(P) ®:F a(Q ®c R)

is commutative for all P, Q, R in C.

I'P·Q®c R --?

IpP®CQ,R

a(P®c Q®c R)

Let FK be the category of functors from OK to itself. The composition law defines a structure of a strict monoidal category on F. On the other hand, the category C of finite dimensional g-modules also has a natural structure of a monoidal category. Therefore it is natural to ask whether the functor P - @P from C to FK can be equipped with a monoidal structure p. In other words, one can ask whether there exists a natural

494 D. Kazhdan

isomorphism D:M,V,W : (M®P)®Q ~ M®(P ® Q) for all M in 0"" P, Q in C such that the corresponding diagrams are commutative. The answer is negative. Moreover the following result is true.

Lemma 8. Assume that K, is a rational number such that the category 0", is rigid and the nominator of K, is bigger than 1. Then there exists M, P, Q as above such that the g-modules (M®P)®Q and M®(P®Q) are

not isomorphic.

Proof. As follows from [KL] there exists a Weil module V~ such that the tensor product V~®V~ is not a sum of Weil modules. Take M = Vo, P = Q = Va,l' Then (M®P)®Q is isomorphic to V~®V~ while the g­module M®(P®Q) is isomorphic to a direct sum of modules M®Vb,l = Vb' bE N1 . Lemma 8 is proved. •

We see that the functor V -+ ®V from a monoidal category C to a monoidal category :F cannot be equipped with any monoidal structure. But it possesses a natural "meromorphic monoidal structure."

To give a definition of a "meromorphic monoidal structure" we have to introduce some notations.

For any A E C* we denote by f -+ f>. the automorphism of the ring C[€, e l ] such that €>. = A€ and we denote by cP>. the linear transformation of g such that cp>.(n.) = n. and CP>.(f ® c) = f>. ®c, f E C[€, ell, c E g. It is easy to see that cP>. is a Lie algebra automorphism of g and cP>. OCP>,' = CP>'>"

for all A, A' E C*. The automorphisms cP>. of g induce the automorphisms V -+ V>. of the

category V, (p, V) -+ (P>., V>.) where V>. is equal to Vasa linear space and p>.(x) = p(cp>.(x» for all x E g. Of course for any P in C, A E C* the g-module P>. lies in C.

Lemma 9. For any M in 0", and P in C there exists a locally free bundle A(M, P) of g- modules over C* with a fiber at A E C* meromorphic to M®P>..

Analogously for any M in 01<, P, Q in C there exist natural locally free bundles A(M, P, Q), 8(M, P, Q) of g-modules over C* x C* with fibers at (A!, A2) isomorphic to (M®P>.) )®Q>'2 and M®(P>.) ®Q>'2) correspondingly.

Theorem 4. There exists a natural isomorphism

defined for real A!, A2 such that 0 < Al « A2 < 00 such that

a) J.l>'),>'2 is real analytic in the area 0 < Al « A2 < 00 and extends to a multi-valued analytic isomorphism between A(M, P, Q) and 8(M, P, Q)

Meromorphic Monoidal Structures 495

on the complement to a finite number of hyperplanes {AdA2 = c}, c E C*.

b) For any triple 0 < Ai « A2 « 1 and any M in 0"" P,Q,R in C the diagram

«M®P>'1)®Q>'2)®R

IJ1. ® id

(M®(P>'l ® Q>'2»®R

is commutative.

id®p. --+ (M®P>'1)®(Q>'2 ® R)

IJ1.

Such an isomorphism J1.>'1,>'2 we call a meromorphic monoidal structure on the functor 0: : P -+ ®P. As it is shown in [KL] this meromorphic monoidal structure on 0: is very much related to the existence of a braided structure on 0",. It is clear how to quantize the categories 0", and C - they become categories Oz,q and Cq of modules over the quantized affine algebra Uq(g). It will be shown in our joint paper with Y. Soibelman that in the case when iqi < 1 it is possible to define the functor 0: : P -+ ®P from Cq to the category :Fz,q of functors from Oz,q to itself and a meromorphic monoidal structure J1. on 0:. It would be very interesting to find how to interpret the existence of the pair (0:, J1.) purely in terms of the category

Oz,q'

References

[KL] D. Kazhdan and G. Lusztig, Tensor structures arising from affine Lie algebras, I-IV, JAMS I, JAMS 6: 4 (1993), 905-948; II, JAMS 6:4, (1993), 949-1011; III, JAMS 7: 2 (1994), in print; IV, JAMS 7: 2 (1994), in print.

[FR] LB. Frenkel and N. Yu. Reshetikhin, "Quantum affine algebras and holonomic difference equations," to appear in the Proceedings of the XXth International Conference on Differential Geometric Methods in Theoretical Physics.

Department of Mathematics, Harvard University, Cambridge, MA 02138, USA.

Received February 1, 1994

The Nil Hecke Ring and Singularity of Schubert Varieties

Shrawan Kumar

Dedicated to Professor Bertram Kostant

Introduction

Let G be a semi-simple simply-connected complex algebraic group and T c B a maximal torus and a Borel subgroup respectively. Let ~ = Lie T be the Cartan subalgebra of the Lie algebra Lie G, and W := N(T)IT the Weyl group associated to the pair (G, T), where N(T) is the normalizer ofT in G. We can view any element w = w mod T E W as the element (denoted by the corresponding German character) III of G I B, defined as III = wB. For anyw E W, there is associated the Schubert variety X w := BwBIB c GIB and the T-fixed points of Xw (under the canonical left action) are precisely Iw := {tl : v E Wand v ~ w}.

We (together with B. Kostant) have defined a certain ring Qw(T) (which is the smash product of the group algebra Z[W] with the W -field Q(T) of rational functions on the torus T ) and certain elements Yw E Qw(T) (for any w E W). Expressing the elements Yw in the {Ov}vEW basis:

we get the matrix B = (bw-l,v-1)w,vEW with entries in Q(T) (cf. §2.1). Analogously, we defined the nil Hecke ring Qw ( which is the smash product of the group algebra Z[W] with the W -field Q(~) of rational functions on the Cartan subalgebra ~) and certain elements Xw E Qw. Writing

we get another matrix C = (CW-l,V-1)W,vEW with entries in Q(~) (cf. §2.4). We prove that the formal T-character of the ring of functions on the

scheme theoretic tangent cone Tu(Xw) (for any tl E Iw) is nothing but *bw-l,v-1 (cf. Theorem 2.2), where * is the involution of Q(T) given by e>' 1--+ e->'. This sharpens a result due to Rossmann [R]. In fact this work of Rossmann, and our own work with B. Kostant on the equivariant K-theory of flag varieties, motivated our current work.

498 s. Kumar

The next principal result of our paper is a necessary and sufficient condition for a point b E Xw to be smooth, in terms of the matrix entry CW-l,u-1 (see Theorem 4.3 (b), for a precise statement).

It should be mentioned that the elements Cw-l,U-1 (as well as bw-l,u-1)

are defined combinatorially and admit closed expressions (cf. Lemma 2.7). The nil Hecke ring approach to singularity, developed in this paper, is applied to some specific examples discussed in Proposition (4.5) and Section (5).

The detailed proofs of the results announced in this article will appear elsewhere.

There are other criteria for smoothness due to Lakshmibai-Seshadri (for classical groups), Ryan (for SL(n)), ... ; and for rational smoothness due to Kazhdan-Lusztig, Carrell-Peterson, Jantzen, ... ; and by works of Deodhar and Peterson, rational smoothness implies smoothness for simply­laced groups. It may be mentioned that our criterion for smoothness (as in Theorem 4.3(b» is applicable to all G uniformly, in contrast to the above­mentioned criteria for smoothness.

Acknowledgements. I am grateful to W. Rossmann, D. Peterson, and J. B. Carrell for explaining to me their (partly unpublished) works. I also thank J. Wahl for some helpful conversations, and the referees for some references. This work was partially supported by the NSF grant No. DMS-9203660 and was written while the author was visiting Tata Institute of Fundamental Research, Bombay, hospitality of which is gratefully acknowl­edged.

1. Notation

Let G be a semi-simple simply-connected complex algebraic group, and let B be a fixed Borel subgroup and T c B a maximal torus. Let B- be the (opposite) Borel subgroup such that B- n B = T. We denote by U (resp. U-) the unipotent radical of B (resp. B-). Let g, b, b-, U, u-, ~ be the Lie algebras of the groups G, B, B-, U, U-, T respectively. Let 6 C ~* (resp. 6+) denote the set of roots for the pair (G,T) (resp. (B,T». Let {a1. ... , an} be the set of simple roots in 6+ and let {a ¥ , ... , a~} be the corresponding (simple) coroots (where n = rank G).

Let W := N(T)jT be the Weyl group (where N(T) is the normalizer of T in G) of G. Then W is a Coxeter group, generated by the simple reflections {rl,··· ,rn } (where ri is the reflection corresponding to the sim­ple root ai). In particular, we can talk of the length f(w) of any element wE W. We denote the identity element of W bye.

Let ~z := {oX E ~* : oX(an E Z, for all i} be the set of integral weights and D:= {oX E ~z : oX(an ~ 0, for all i} the set of dominant

Nil Heeke Ring 499

integral weights. For any A E D and w E W, we denote by V (A) the irreducible representation of G with highest weight A, and Vw(A) is the smallest B-submodule of V(A) containing the extremal weight vector ew >. (of weight wA). Let R(T) := Z[X(T)] be the group algebra of the character group X(T) of the torus T. Then {e>' hE~z are precisely the elements of X(T). Let Q(T) be the quotient field of R(T). Clearly W acts on Q(T) and moreover Q(T) admits an involution * (Le. a field automorphism of order 2) taking e>' f--+ e->'.

For any wE W, the Schubert variety Xw is by definition the closure of BwBIB in GIB under the Zariski topology (where the notation BwBIB means BwBIB for any representative w of w in N(T». Then Xw is an irreducible (projective) subvariety of GIB of dimension l(w). By the Bruhat decomposition, any II such that v :5 w belongs to X w , where :5 is the Bruhat (or Chevalley) partial order in W. The Schubert variety Xw is clearly B-stable (in particular T-stable), under the left multiplication of B on GIB. The T-fixed points of Xw are precisely Iw .- {tl : v E Wand v:5 w}.

2. Character of the ring of functions on the tangent cone of Xw

We follow the notation as in §1.

(2.1) Definitions. (a) For any local ring R with maximal ideal m, define the graded Rim-algebra:

gr R := 2:mn Imn+!. n~O

Let X be a scheme of finite type over an algebraically closed field and let x be a closed point of X. Then the tangent cone Tx(X) of X at x is, by definition (cf. [M, Chapter 3, §3j), Spec (gr Ox), where Ox = Ox,x is the local ring at x EX.

~

(b) Let R(T) be the set of all the formal sums 2: n>.e>', with arbitrary eAEX(T)

n>.~ E Z (we allow infinitely many of the n~s to be non-zero). Even though

R(T) is not a ring, it has a canonical R(T)-module structure (got by the ~ ~

multiplication). We define the Q(T)-module Q(T) as Q(T)®R(T) R(T). ~

Since Q(T) is a flat R(T)- module, Q(T) canonically embeds in Q(T).

(c) AT-module M is said to be a weight module if M = EeeAEX(T)M>., where M>. := {m EM: tm = e>'(t)m} is the A-th weight space. A weight

500 S. Kumar

module M is said to be an admissible T-module if dim M>. < 00, for all e>' E X(T).

For any admissible T-module M, one can define its formal charac-

ter ch M:= L (dim M>.) e>' as an element of Rfr). e>'EX(T)

(d) The ring Q(T)w ([KK2, Section 2]): Let Q(T)w be the smash product of the W -field Q(T) with the group algebra Z[W], i.e., Q(T)w is a free right Q(T)-module with basis {Ow}wEW and the multiplication is given by:

(1) (OW1Qt}'(Ow2q2) = OW1W2(W2"lQt)Q2, for Qt,Q2 E Q(T) and Wt,W2 E W.

For any simple reflection ri, 1 ::; i ::; n, define the element Yr, E Q(T)w by:

(2)

Now, for any wE W, define Yw E Q(T)w by

(3)

where w = rh ... rip is a reduced decomposition. By [KK2, Proposition 2.4], Yw is well defined. Write

(4) v

for some (unique) bw-l,v-1 E Q(T).

Since t:l E Xw is fixed under the action of T (cf. §1), the local ring Ou,xw at t:l E Xw is canonically a T-module. The following result was obtained (and privately circulated) by the author in 1987.

(2.2) Theorem. Take any v::; w E W. Then gr Ou,xw is an admissible T - module and moreover

~ ~

as elements of Q(T), where ch (which is an element of R(T» is to be thought ~ ~

of as the element 1® ch of Q(T):= Q(T)®R(T) R(T). In particular, ch (gr Ou,xw ) E Q(T).

For any variety X over C, we denote by qX] the ring of global regular functions X -+ C. The proof of the above theorem makes use of the follow­ing lemma. The second part of the following lemma is easily obtained by

Nil Heeke Ring 501

using the first part and [BGG, Theorem 2.9], and (as one of the referees has pointed out) the first part was proved by Andersen and Cline-Parshall-Scott in 1980.

(2.3) Lemma. Given any f E qu-J, there exists a large enough A E D {i.e. A(an » 0, for all the simple coroots an and (J E V(A)* such that

f(g) =< (J,ge>. >, for 9 E U-,

where e>. is a non-zero highest weight vector of V(A). Moreover, for any v ~ w E W, f vanishes on (v- 1 EwE) n U- {:}

v«(J) E [V(A)jVw(A)]*.

After the following definitions, we give some of the corollaries of The­orem (2.2).

(2.4) Definitions. (a) For any £ E Z+ := {O, 1, 2,···} and any a = E n>.e>' E R(T), denote by (a)e = En>. ~: E Se(~*), where Se(~*) is the space of homogeneous polynomials of degree £ on~. Further, denote by [a] = (a)eo where £0 is the smallest element of Z+ such that (ak =1= O. (If a itself is 0, we define [a] = 0.) Now for q = i E Q(T), where a, bE R(T), we

define [q] = ~ E Q(~) (the quotient field of the symmetric algebra S(~*». Clearly [q] is well defined.

When q =1= 0 and deg [a] ~ deg [b], we say that q has a pole (at the identity e) of order = deg [b]- deg [a]. It is easy to see that bw-I,V-1 (cf. (4) of §2.1), when non-zero, has a pole of order ~ £(w).

(b) The nil Hecke ring Qw ([KKl, §4]): Let Qw be the smash product of the W -field Q(~) with the group algebra Z[W], with the product given by the same formula (1) in § 2.1. For any simple reflection ri, 1 ~ i ~ n, define xr, E Qw by x r, = -(Ori + Oe) ~i' Now, for any w E W, define Xw = Xril ... x rip ' where w = ri l ••• rip is a reduced decomposition. The element Xw is well defined by [KKl, Proposition 4.2]. Write, as in [KKl, Proposition 4.3],

Xw = LCw-l,v-1 ov, for some ( unique) Cw-l,v-l E Q(~) . v

(2.5) Corollaries. For any v,w E W: {aj bw,v =1= 0 if and only if v ~ w, and in this case it has a pole of order exactly equal to £( w).

Further, ( IT (1 - ef3»bw ,v E R(T) . f3EA+

{bj [*bw-l,v-l] = Cw-l,v-l; and hence for any v ~ w, [ch (gr Op,xw )] = Cw-l,v- l , as elements ofQ(~).

502

In particular, cw,v f:. 0 if and only if v ::::; w.

Further ( II (J)cw ,v E S(~*). f3ED.+

s. Kumar

(2.6) Remark. The (b)-part of the above corollary is due to Rossmann [R, §3.2j. In fact this motivated our theorem (2.2).

The following lemma gives an expression for bw,v (and cw,v) and can easily be proved by using the definitions.

(2.7) Lemma. Fix any v ::::; w E W , and take a reduced decomposition w = ril ... rip. Then

Similarly

where both the sums run over all those (1"1. ... , €p) E {O,l}P satisfying r7: ... r:: = v. (The notation r? means the identity element.)

3. Ring of functions on the tangent cone -the graded algebra structure

(3.1) For any>. E D, the (finite dimensional) G-module V(>') admits a filtration {.rp(>')}P~o as follows:

Let {Up(u-)}p~o be the standard filtration of the universal enveloping algebra U(u-), where we recall that Up(u-) is the span of the monomials Xl·· ·Xm for Xi E u- and m::::; p. Now set

where eA is any non-zero highest weight vector in V(>.). Define a partial order ~ in D by >. ~ J-l {:} J-l- >. ED. For any>., () ED,

define a T-equivariant map 1TA,O : V(>'+() --+ V(>')@Co as the composition

V(>' + () --+ V(>') @ V«() --+ V(>.) @ Co,

where Co is the one dimensional T-module corresponding to the character eO, the first map is the unique G-module map taking eA+o 1-+ eA @ eo and the second map is induced from the projection V«() --+ Co onto the ()-weight space.

Nil Heeke Ring 503

For any v:::; W E W, the T-module map 7rA,8 gives rise to aT-module map (for any ). ~ J.L and p ~ 0 ; taking () = J.L - ).)

i.e.,

[ Fp().) n V-I Vw ().) ] * { Fp-l().) nv-1Vw ().) ®CA}>.ED

is a directed system of T-modules.

(3.2) Theorem. Fix v :::; w E W, and let gr (O"v-1x,J = ffip 2:o grp(O"v-1x,J be the graded algebra associated to the local ring O"tI-1xw at the point e of v-I Xw {cf. § 2.1}. Then, for any p ~ 0, there is a certain natural T - equivariant isomorphism

For any variety X and a closed point x E X, let Zx(X) denote the Zariski tangent space of X at x. For any subvariety Y C G j B containing the base point e, we get the induced inclusion Z,(Y) L-+ Z,(GjB). But Z,(GjB) can canonically be identified with u- (since U- is an open neigh­borhood around e in GjB), in particular, Z,(Y) can canonically be viewed as a subspace of u-. For any a E 6, let ro< E W be the reflection defined by r Q ().) =).- < ).,av > a.

As a consequence of Theorem (3.2), we obtain the following. (As I am informed by one of the Referees, the following corollary also follows from [C, Theorem G (2)].)

(3.3) Corollary. Let G be simply-laced and take v :::; w E W. Then the tangent cone T,(v- l Xw) is non-reduced {as a scheme} if there exists a -a E ~+ such that vrQ 1:. w but Xo< E Z,(v- 1 X w ), where X Q is a non-zero root vector corresponding to the {negative} root a.

4. Nil Heeke ring and smoothness

We follow the notation as in § 1. Recall the definition of the elements Cw-l,v-1 E Q(~) from § 2.4. For any v :::; w, define

Sw-1,v-1 = {.8 E ~+ : v- 1r{3 :::;w-1 }.

504 S. Kumar

Clearly 1 Sw-1,v-1 1=1 Sw,v I, where 1 S 1 denotes the order of the set S. Although we do not need, let us recall a very interesting conjecture of Deodhar (proved by Carrell- Peterson [C], Dyer [D], and Polo [P] ) asserting that 1 Sw-1,v-1 I~ few).

Even though the following proposition follows immediately by com­bining our Corollary 2.5 (b) with [D, Proposition on page 573], we give a different (geometric) proof (as that proof is crucially used in the proof of Theorem 4.3 (b) ).

(4.1) Proposition. Let v $ w E W. Then 1 Sw-1,v-1 1= few) <=> [ch (gr O",xw )] = d (_1)iCw l-lCv l II {3-1, for some d E Co

(3ESw -l,v-1

(4.2) Remark. In fact, when the equivalent condition as above is satisfied, d is a positive integer.

We come to the following main result of this article. Recall the defini­tion of a rationally smooth point y in a vari~ty Y from [KL, Appendix].

(4.3) Theorem. Fix v $ w E W. (a) The point \) E Xw is mtionally smooth <=>

For all v $ () $ w, we have

(1)

for some constants do E C. (b) The point \) E Xw is smooth <=>

C - (_1)lCwl-lCv l w-1,v-1 -

{3-1,

II

(4.4) Remarks. (1) The (a) part of the above theorem follows immedi­ately by combining a result of Dyer [D, Proposition on page 573] with a result of Carrell-Peterson [C, Theorem E].

(2) In the case (a) as above (i.e. if \) E Xw is rationally smooth), the constant do is in fact a positive integer.

(3) There are some examples of \) E Xw (where Xw is even a codimension one Schubert variety in G/B) such that CW-l,V-1 satisfies the condition (1) of the above theorem, but \) is not a rationally smooth point of Xw. In particular, to check the rational smoothness of a point \) E X w , it is not sufficient (in general) to check the validity of the condition (1) only for () = v.

Nil Heeke Ring 505

(4) It is a result of V. V. Deodhar that any rationally smooth Schubert variety is in fact smooth for G = SL{n). This result has recently been extended for any simply-laced G by D. Peterson. As is well known, this result is false in general for non simply-laced G.

As an immediate corollary of Theorem (4.3) and Lemma (2.7), we ob­tain the following result determining the singular locus of all the Schubert varieties in the case of any rank two group. I believe it should be well known, but I did not find it explicitly written down in the literature. We follow the indexing convention as in Bourbaki [BJ.

(4.5) Proposition. The following is a complete description of the singular loc'us in the case of rank two groups: Case I. G = A2 : In this case all the six Schubert varieties are smooth. Case II. G = C2 : There are, in all, eight Schubert varieties. Out of these only Xrlr2rl is singular and it has singular locus = X r1 .

Case III. G = G2 : There are, in all, twelve Schubert varieties. Following is the complete list of singular ones and their singular loci:

Singular locus

(1) Xrlr2rl X r1

(2) XrlT2TtT2 X r1r2

(3) Xr2rlr2rl X r2r1

(4) Xrlr2rlr2rl Xrlr2rl

(5) Xr2rlr2rlr2 X r2

5. Examples of codimension one Schubert varieties in G I B

Let Wo be the longest element of the Weyl group W (of G). As is well known, the codimension one Schubert varieties in G I B are precisely of the form Xw , where w = wori for a simple reflection rio In particular, the number of such Schubert varieties in GIB is equal to n := rank G. We denote the Schubert variety Xwori (1 :::; i :::; n) by Xi. Let Xi E ~z be the i-th (1:::; i :::; n) fundamental weight, defined by Xi{a'j) = 8i ,j'

(5.1) Lemma. Fix any 1 :::; i :::; n. Then for any v E W such that v:::; Wori,

where [ J is as in § 2.4 (aJ.

506 S. Kumar

By virtue of the above lemma, we get the following strengthening of Theorem 4.3 (b) for the Schubert varieties Xi.

(5.2) Proposition. Let Xi (1 ~ i ~ n) be a codimension one Schubert variety as above. Then, for any v ~ wori E W, the following are equivalent:

(1) U E Xi is smooth.

(2) cr,wo ,u-1 (_l)N-I-l(u) {31" '~N-l' for some positive roots {.Bll · .. ,.BN-d (where N = dim G / B).

(3) Xi - V-IWoXi is a root.

(5.3) Remark. If U E Xi is smooth, then the set {.Bt. . .. , .BN-I}, as in (2) above, coincides with the set Sr,wo,v-1 (by Theorem 4.3 (b».

We assume that G is a simple group. Combining Theorem (4.3), Propo­sition (5.2), Lemma (5.1), and Corollary 2.5 (b), we get the following:

(5.4) Proposition. (a) The following is a complete list of codimension one Schubert varieties Xi which are smooth:

(al) An (n ~ 1)

(a2) en (n ~ 2)

i = 1,n

i = 1.

(b) The following is a complete list of codimension one Schubert varieties Xi, which are rationally smooth but not smooth:

(bI) Any non-smooth Xi in any group G with rank ~ 2 (b2 ) Bn (n> 2) : i = 1.

(5.5) Remark. I am informed by one of the Referees that the (a) part of the above proposition, as well as the equivalence of (1) and (3) in Propo­sition (5.2) for v = e was contained in an earlier longer version of [C]. Of course (b l ) is very well known, and the example (b2 ) was known to be rationally smooth by some work of Boe [Bo].

References

[B] Bourbaki, N., "Groupes et algebres de Lie, Chap. IV-VI," Hermann, Paris, 1968.

[BGG] Bernstein, I. N., Gel'fand, I. M., and Gel'fand, S. I., Schubert cells and cohomology of the spaces G / P, Russian Math. Surveys 28(1973), 1-26.

[Bo] Boe, B. D.: Kazhdan-Lusztig polynomials for hermitian symmetric spaces, Trans. A.M.S. 309(1988), 279-294.

Nil Hecke Ring 507

IC] Carrell, J. B., The Bruhat graph of a Coxeter group, a conjecture of Deodhar, and rational smoothness of Schubert varieties, Proc. Sympo. Pure Math. vol. 56 (1994), edited by W. J. Haboush and B. J. Parshall, 53-61.

ID] Dyer, M. J., The nil Hecke ring and Deodhar's conjecture on Bruhat intervals. Invent. Math. 111 (1993), 571-574.

[J] Jantzen, J. C., " Moduln mit einem hochsten Gewicht," LNM 750 (1979), Springer-Verlag, Berlin-Heidelberg-New York.

IKKl] Kostant, B., and Kumar, S., The nil Hecke ring and cohomology of GIP for a Kac-Moody group G, Advances in Math. 62 (1986), 187-237.

[KK21 Kostant, B., and Kumar, S., T-equivariant K -theory of generalized flag varieties, J. Ditt. Geometry 32 (1990), 549-603.

[KLJ Kazhdan, D., and Lusztig, G., Representations of Coxeter groups and Hecke algebras, Invent. Math. 53 (1979), 165-184.

[LS] Lakshmibai, V., and Seshadri C. S., Singular locus of a Schubert variety, Bull. A.M.S. 11 (1984), 363-366.

[M] Mumford, D., "The red book of varieties and schemes," LNM 1358 (1988), Springer-Verlag.

IP] Polo, P., On Zariski tangent spaces of Schubert varieties, and a proof of a conjecture of Deodhar, Preprint (1993).

[R] Rossmann, W., Equivariant multiplicities on complex varieties, Asterisque 173-174 (1989), 313-330.

[Ry] Ryan, K. M., On Schubert varieties in the flag manifold of SL(n, q, Math. Annalen 276 (1987) , 205-224.

Department of Mathematics, University of North Carolina, Chapel Hill, N.C. 27599-3250

Received February 17, 1994

Some Classical and Quantum Algebras

Bong H. Lian and Gregg J. Zuckerman

ABSTRACT. We discuss the notion of a Batalin-Vilkovisky (BV) algebra

and give several classical examples from differential geometry and Lie the­

ory. We introduce the notion of a quantum operator algebra (QOA) as a

generalization of a classical operator algebra. In some examples, we view

a QOA as a deformation of a commutative algebra. We then review the

notion of a vertex operator algebra (VOA) and show that a vertex operator

algebra is a QOA with some additional structures. Finally, we establish a

connection between BV algebras and VOAs.

1 Introduction

In reference [19], the authors established a precise and general connec­

tion between two types of algebras which are well known in contemporary

mathematical physics: Batalin-Vilkovisky algebras (BV algebras) and ver­

tex operator algebras (VOAs). BValgebras, although implicit in modern

mathematics, arose for the first time explicitly in the context of BV quan­

tization of classical field theories. VOAs arose for the first time in the

parallel contexts of two dimensional conformal quantum field theory [1] and the mathematical theory of monstrous moonshine [2] [5].

The present paper may be regarded as a very brief mathematical in­

troduction to both BV algebras and VOAs. We have attempted to relate

both types of algebras to objects and constructions in differential geometry,

supergeometry, Lie theory, commutative algebra, homological algebra and

operator algebras.

In section 2, we point out that the algebra of differential forms on a semi-Riemannian manifold is naturally a BV algebra. Thus, BV algebras

were implicit long ago in the absolute tensor calculus and in general rela­

tivity. Likewise, the algebra of exterior forms on a finite dimensional Lie

11991 Mathematics Subject Classification. Primary 81 T70, 17B68. 2G.J.Z. is supported by NSF Grant DMS-9008459 and DOE Grant DE­

FG0292ER25121.

510 B. H. Lian and G. J. Zuckerman

algebra, which is endowed with a nondegenerate symmetric bilinear form

(or Hermitian form), is naturally a BV algebra.

As a pedagogical device in our discussion of VOAs, we introduce in section 3 the very simple and abstract notions of quantum operators and

quantum operator algebras (QOAs). Physicists have shied away from such

abstractions, but we have no such inhibitions. We have found that many of

the fundamental ideas and formulas in conformal field theory can be very

naturally explained in the setting of QOAs.

In order to connect QOAs to VOAs, we present in section 4 a quick

discussion of three points of view on commutative algebras. We then review

the definition of a VOA in section 5, and discuss this definition from three

new points of view that parallel our previous discussion in section 4. The

main new result of the present paper is Theorem 5.6, which reformulates the

notion of a VOA (without a Virasoro quantum operator) as a particular and

remarkable type of QOA, which we call a creative QOA. We then present

two fundamental constructions of creative QOAs. The second construction

is a reformulation of some work of Igor Frenkel and Yongchang Zhu [7]. The

approach of the present paper was strongly influenced by the Frenkel-Zhu

article.

In the final section of this paper (see Theorems 6.3 and 6.4), we restate

the main theorems of [19] as theorems about conformal QOAs (which are

now equipped with a Virasoro quantum operator). In brief, we employ

the BRST precedure [14] [3] [4] to formulate a cohomological construction

of a BV algebra that starts from a choice of conformal QOA with central

charge twenty-six. Our main example requires one ingredient: a simple Lie

algebra over the complex numbers. Our construction goes through without

a hitch because of the remarkable fact that there does not exist a simple

Lie algebra having dimension twenty-six.

Many other known constructions in string theory can in fact be for­

mulated in the language of QOAs and the BRST procedure (see also

[24][26][16][17][18]). The results on BV algebras in [19] have been applied

in a recent paper by Greg Moore [20].

The present paper grew out of a pair of lectures presented by the au­

thors in October 1993 to the University of North Carolina Mathematics

Department. We thank James Stasheff for the opportunity to give these

lectures. The authors are very pleased to dedicate this paper to Professor

Some Classical and Quantum Algebras 511

Kostant on the occasion of his sixty-fifth birthday.

2 Batalin-Vilkovisky Algebras

Let A * be a Z graded commutative associative algebra. For every a E

A, let la denote the linear map on A given by the left multiplication by a. Recall that a (graded) derivation d on A is a homogeneous linear operator

such that [d, la]- lda = 0 for all a. A BV operator [25][23][11] A on A* is

a linear operator of degree -1 such that:

(i) A2 = 0;

(ii) [A, la] - lLla is a derivation on A for all a, ie. A is a second order

derivation.

A BV algebra is a pair (A, A) where A is a graded commutative al­

gebra and A is a BV operator on A. The following is an elementary but

fundamental lemma:

Lemma 2.1 [11]{21} Given a BV algebra (A, A), define the BV bracket

{,} on A by:

Then {,} is a graded Lie bracket on A of degree -1.

By property (ii) above, it follows immediately that for every a E A, {a, - } is a derivation on A. Thus a BV algebra is a sort of odd Poisson algebra

which, in mathematics, is also known as a Gerstenhaber algebra [9][10].

2.1 Some Classical Examples

Let M be an n-dimensional smooth manifold with a fixed volume form

w. Let C*(M) be the deRham complex. Let V*(M) be the algebra of

polyvector fields - ie. the exterior algebra on the smooth vector fields

Vect(M) over the the ring of smooth functions on M. There is a canonical

degree reversing linear isomorphism iw : V*(M) ---t cn-*(M) given by

the contraction with w: v 1--+ iw(v). Conjugating the deRham differential

d by this isomorphism, we obtain a square zero degree -1 operator Aw = i;ldiw on V*(M). In local coordinates xi for which the volume form is

512 B. H. Lian and G. J. Zuckerman

w = dx1 /\ .•• /\ dxn, we have

8 8 b.w = "'""' -8 . -8 . * , L...J x' x,

i

(2.1)

where Xi* denotes the generator a~. in V1(M). It is evident that b.w is a

second order derivation on V*(M), hence making (V*(M), b.w ) into a BV algebra [23].

Clearly the above construction generalizes to spaces in other categories: algebraic, holomorphic etc.

The construction may be viewed slightly differently: we can regard

V* (M) as the commutative superalgebra of functions on nT* M, the cotan­

gent bundle of M with the fibers made into odd supervector spaces. The

BV bracket in V* (M) turns out to be equal to the odd Poisson bracket as­

sociated to the canonical odd symplectic two-form on nT* M. The bracket

is also known as the Schouten bracket.

We should mention an important and well-known application of the

algebra V*(M). Let P be a bivector field on M. We can always construct

a bracket operation on the function algebra Coo (M) by the formula

[f,g]p = Lp(df /\ dg) (2.2)

where Lp denotes the contraction of P against a two-form. The question

is: when does the new bracket [,] give rise to a Lie algebra structure on

COO(M) (hence a Poisson algebra)?

Proposition 2.2 The bracket [,] satisfies the Jacobi identity iff the BV

bracket {P, P} = o.

To discuss our second example, let's assume that the volume form w above comes from a metric 9 on M. Let d* be the formal adjoint of the

deRham differential d, relative to the metric g. Note that d* is a square

zero degree -1 operator on C* (M) . Choose local coordinates so that the

Riemannian volume form is w = dx1 /\ ... /\ dxn. Then once again d* has

exactly the same form (2.1) where now Xi* denotes the generator dx i in

the algebra C*(M). Therefore d* is also a second order derivation making

(C*(M),d*) into a BV algebra which depends on g.

It turns out that this BV algebra is isomorphic to (V* (M), b.w ). Locally

this isomorphism is determined by dxi 1-+ gij a~J. The metric ensures that

this is well-defined globally.

Some Classical and Quantum Algebras 513

We now come to our third example, which arises in Lie theory. Let g be

any Lie algebra and A * g be its exterior algebra. Let 6 be the Lie algebra

homology differential on A * g

6(Xl /\ ... /\Xp) = L(-l)i+i[Xi,Xi ] /\X1 /\ .• . Xi •• .Xi ··· /\ XV" (2.3) i<i

Then by direct computation we verify that 6 is a BY operator on the exterior

algebra.

3 Quantum Operator Algebras

Let V be a Z doubly graded vector space V = Elwn(m] such that

V[m] = 0 for all but finitely many negative m's. The degrees of a homoge­

neous element v in vn [m] will be denoted as Ivl = n, IIvll = m respectively.

In physical applications, Ivl will be the fermion number of v. In conformal

field theory, IIvll will be the conformal dimension of v.

Let z be a formal variable with degrees Izi = 0, IIzll = -1. Then it makes sense to speak of a homogeneous (biinfinite) formal power series

a(z) = EnEZ a(n)z-n-l of degrees la(z)l, lIa(z)1I where the coefficients

a(n) are linear maps in V with degrees la(n)1 = la(z)l, lIa(n)1I = -n - 1 + lIa(z)lI. Note then that the terms a(n)z-n-l indeed have the same degrees

la(z)l, lIa(z)1I for all n. We call a finite sum of such homogeneous series a(z) a quantum operator on V, and we denote the linear space of quantum

operator as QO(V).

Clearly it does not makes sense in general to multiply two quantum

operators a(z), b(z) pointwise. However the space Lin(V) of graded linear

maps in V can be viewed as a subspace of QO(V) where a linear map A

is regarded as the constant series E A6n,_lZ-n-1. The space Lin(V) is an

associative algebra in a canonical way. Is there a natural way to extend the

product in Lin(V) to all of QO(V)? One such (nonassociative) extension

is known as the Wick product, defined as:

: a(z)b(z) := L a(n)z-n-1b(z) + (_l)lallblb(z) L a(n)z-n-l. (3.4) n<O n~O

It is evident that when restricted to Lin(V), the Wick product coincides

with the natural product in Lin(V).

514 B. H. Lian and G. J. Zuckerman

Here we introduce a notation for iterated Wick products which we will

use later. Let a1(z), ... , an(z) be quantum operators. Their Wick product is defined inductively as follows:

: a1(z)··· an(z) :~: al(z)(: a2(z)··· an(z) :) : (3.5)

There is in fact a family of products (of which the Wick product is one)

which measure formally the singularity of the formal product a(z)b(w) as

z "approaches" w.

Definition 3.1 For each integer n we define a product on QO(V):

a(w)onb(w) = Resza(z)b(w)tz,w(z-w)n_( -l)la ll bIReszb(w)a(z)tw,z(z-w)n

(3.6) where

(3.7)

To see that the above products are well-defined, take an element v in V and consider first

Resza(z)v tw,z(z - w)n = L ( ~ ) (_l)n- i wn- i a(i)v. (3.8) i~O ~

This is a finite sum because a(i)v = 0 for all but finitely many positive i.

So Reszb(w)a(z)v tw,z(z - w)n makes sense for any v and hence defines an

element in QO(V). Similarly for Resza(z)b(z)v tz,w(z - w)n. We note that

a(z) 0_1 b(z) is nothing but the Wick product: a(z)b(z) :.

Proposition 3.2 For a(z), b(z) in QO(V), the following equality of formal

power series in two variables holds:

a(z)b(w) = L a(w) On b(w)tz,w(Z - w)-n-l+ : a(z)b(w) : . (3.9) n~O

In this sense: a(z)b(w) : is the nonsingular part of the operator product

expansion (3.9) (see [1]), while a(w) On b(w)tz,w(z - w)-n-1 is the polar

part of order -n - 1. We note that for n < 0, we have

1 a(z) On b(z) = (-n _ 1)! : a-n- 1a(z) b(z) : (3.10)

Some Classical and Quantum Algebms 515

where 8 = tz' For n 2: 0, we have

a(z) On b(z) = [(to ( : ) a(m)( _z)n-m), b(z)). (3.11)

Lemma 3.3 Let A be a homogeneous linear operator on V. Then the

commutator [A, -) is a graded derivation of each of the products On.

Note that for n = 0, we have a(z) 00 b(z) = [a(O),b(z)). As a corollary,

we have

Proposition 3.4 For any a(z), b(z), c(z) in QO(V) and n integer, we have

a(z) 00 (b(z) On c(z)) = [a(z) 00 b(z)) On c(z) + (-1)lallblb(z) On [a(z) 00 c(z)),

ie. a(z)oo is a derivation of every product in QO(V).

The products On will become important for describing the algebraic and

analytic structures of certain algebras of quantum operators. Thus we

introduce the following mathematical definitions:

Definition 3.5 We say that a(z), b(z) are mutually local if a(z) On b(z) = 0

for all but finitely many positive n.

Definition 3.6 A graded subspace A of QO(V) containing the identity

operator and closed with respect to all the products On is called a quan­

tum operator algebra. A QOA A is called local if its elements are pairwise

mutually local.

We observe that for any element a(z) of a QOA, we have a(z)0_21 = 8a(z).

Thus a QOA is closed with respect to formal differentiation.

Proposition 3.7 Suppose 0 is a local QOA. Then for any a(z), b(z) in

0, the matrix entries of a(z)b(w) are rational.

Proof: Let v be in V and v* in the restricted dual of V. Byeqn (3.9), we have

(v"la(z)b(w)lv) = L(v*la(w)onb(w)lv)(z-w)-n-l+(v*l: a(z)b(w): Iv). n20

(3.12)

516 B. H. Lian and G. J. Zuckerman

Each of the matrix entries, (v*la(w) On b(w)lv} is a Laurent polynomial in

w. If a(z), b(z) are mutually local, then we have only finitely many (z - w)­polar terms in the above expansion. Thus the singular part of the operator

product expansion contributes a rational function of z, w to the matrix

entry. The matrix entry of the Wick product is a Laurent polynomial. 0

We note that none of the products On is associative in general. (We

will return to this point later in an example.) However it clearly makes

sense to speak of the left, right or two sided ideals in a QOA and they are

defined in an obvious way.

We now return to a classical setting to motivate a construction in the

theory of QOAs.

4 Commutative Algebras

Let A be a vector space with a distinguished nonzero element 1. We want to consider the set C(A,l) consisting of commutative, associative

bilinear products m on A such that 1 is the identity element for m. We

want to relate this set to some other sets naturally associated to the based

space (A,l).

Definition 4.1 A translation map for (A, 1) is a linear map Y : A ~

Lin(A), a 1-+ yea), such that the following axioms hold: (l)Y(l) = id (2)Y(a)1 = a (3)Y(a)Y(b) = Y(b)Y(a) = Y(Y(a)b). Let T(A, 1) denote the set of all translation maps for (A, 1).

A translation map Y translates states (elements of A) into operators (el­

ements of Lin(A». For each translation map, we can define a product

my: A®A~A my(a, b) = Y(a)b. (4.13)

Then my is commutative and associative and 1 is the identity element.

Moreover the map Y is an injective homomorphism of algebra (A, my, 1)

into the algebra Lin(A). Conversely if mE C(A, 1), we can define the map Ym : A ~ Lin(A)

Ym(a)b = mea, b). ( 4.14)

Some Classical and Quantum Algebras 517

Then it is clear that Ym is a translation map.

Proposition 4.2 The sets C(A, 1) and T(A, 1) are in bijective correspon­dence via the map m f-4 Ym , and its inverse Y f-4 my.

4.1 Creative Operator Algebras

The above Proposition is of course trivial to prove, but it is a stepping

stone to the following idea. Let Y be a translation map. Let Oy denote

the set of operators Y(A). We know that Oy is a commutative algebra of

linear operators on A and that the map Oy --+ A, Y(a) --+ Y(a)l is a

linear isomorphism.

Definition 4.3 A creative operator algebra for the based space (A, 1) is a commutative subalgebra 0 of Lin(A) such that the map 0 --+ A, x f-4 xl is a linear isomorphism. We call this map the creative map associated to

O. We write CO(V, 1) for the set of creative operator algebras for (V, 1).

It turns out that in the above definition, it is enough to require that the

creative map is surjective.

If 0 is a creative operator algebra, let Yo : A --+ 0 C Lin(A) be the

inverse of the creative map 0 --+ A associated with O. Thus Yo(x1) = x for all x E 0, or equivalently Yo(a)l = a for all a E A. It is easy to check

that Yo is a translation map.

Proposition 4.4 The sets T(A,l) and CO(A,l) are in bijective corre­spondence via the map Y f-4 Oy and its inverse map 0 f-4 Yo.

To gain further insight into creative operator algebras we state:

Proposition 4.5 If 0 is in CO(A, 1), then

(i) 0 is a maximal commutative subalgebra of Lin(A); (i'i) 0 is a complementary subspace to the annihilator Ann(l) of 1 in Lin(A).

5 Vertex Operator Algebras

Recall that the formal variable z is assigned degrees Izl = 0, IIzll = -1.

518 B. H. Lian and G. J. Zuckerman

Definition 5.1 [2][5} A VOA is a doubly graded vector space V = EBVk[l]

with a distinguished element 1 with both degrees equal to zero, and a degree

preserving linear map Y( -, z) : V --t QO(V), satisfying the following conditions:

(i) (Boundedness) For each k, Vk[l] = 0 for all but finitely many negative

1; (ii) (Unit) Y(l, z) = id;

(iii) (Non degeneracy) For any v in V, Y(v, z)l is a formal power series

and

limz->oY(v,z)l = v.

(iv) (Cauchy-Jacobi identity) For any v, v' in V, and integers n, k, 1

Resz_wY(Y(v, z - w)v', w) Lw.z_wznwk(z - w)l

= ReszY(v, z)Y(v', w) Lz.wznwk(z - w)l

-( -l)lv llv'IReszY(v', z')Y(v, z) Lw.zznwk(z - W)I (5.15)

For now we will ignore the so-called Virasoro structure on a VOA. (See

below.)

We want to develop the theory of VOA's as a straightforward but deep

generalization of the theory of commutative algebras. We have developed

three points of view about commutative algebras:

1. Bilinear product on a based space;

2. Translation map for a based space;

3. Creative operator algebra for a based space.

We have demonstrated the elementary equivalence of these three points

of view. Let's sketch informally what we know about these three levels for

VOA theory.

1. A VOA can be described in the following terms:

(i) A bigraded vector space V = EBVk[l] with a distinguished element 10f

degrees 111 = 0 = 11111. We will assume for simplicity that the second degree is bounded from below. (ii) A family of bilinear products m n, n E Z , on V. The degrees of mn

are Imnl = 0, Ilmnll = -n - 1. Thus for any pair of elements a, b in V, mn(a, b) = 0 for all but finitely many positive n. Only m-l is of second degree zero.

(iii) Axioms relating to 1: for any a in V,

a) for any nonnegative integer n, mn(a, 1) = 0

Same Classical and Quantum Algebras

b) m_l(a, 1) = a

c} for any integer n # -1, m n (1,a} = 0

d} m_l(1,a} = a.

519

Thus the operation m_l behaves most classically: 1 is an identity element

for m-l. For the operation mn with n < -1, 1 is only a right identity. For

the operation mn with n > 0, 1 is a universal left and right zero divisor.

(e) Cauchy-Jacobi axiom system: This is a complex system of trilinear

identities for the system of operations m n . Two kinds of terms appear:

mn(a, mp(b, c», mp(mn(a, b}, c} with various permutations of the three el­

ements of V.

Remark 5.2 (i) It would be very difficult at the present time to motivate the Cauchy-Jacobi axiom system in the purely classical setting of systems

of bilinear products. For one thing, it would be difficult to give a simple

and compelling example of an algebra satisfying all of the above axioms. Moreover, the actual known examples of VOA's did not arise in the language of this first level.

(ii) It is already difficult to motivate algebras with more than two bilinear

products. We know that in some sense the operations mn in a VOA fall

into two classes: n nonnegative, and n negative. We also know that in

applications, the two operations, mo and m-l appear most often. However,

the operation ml also occurs in our work on BV operators (see below). Thus

the B V operation in this language is ml (b, v) where b is a distinguished element.

2. Associated to a VOA V is the vertex map, Ym , where m now stands for

the sequence of operations mn in V:

Ym(v,z}v' = Lmn(v,v'}z-n-l. (5.16) n

To hide the direct reference to m, we can introduce the mode operators

v(n} such that Ilv(n)11 = IIvll- n - 1 if v is homogeneous, and such that

mn(v,v'} = v(n)v'. (5.17)

Finally we introduce the vertex operator

Y(v, z} = L v(n}z-n-l (5.18) n

520 B. H. Lian and G. J. Zuckerman

for each v in V. We can write the standard duality axioms for a VOA

in terms of the vertex operators above. We obtain axioms that look like

generalizations of the axioms for a translation map (Definition 4.1).

(i) Y(l, z) = idj

(ii) For any v in V, Y(v, z)l is a formal power series and

limz->oY(v,z)l = v.

(iii) For any v, Vi in V [5][6],

Y( v, z)Y( Vi, Zl) '" (_1)lv 11v'ly( Vi, Zl)y( v, z) '" Y(Y( v, z - Z')V' , Zl),

where the relation", has to be defined carefully in terms of the formal

variable calculus. In particular we need to assume the rationality of the

matrix entries of the three formal expressions above. (See Proposition 3.7.)

Observation: If all the vertex operators Y(v, z) are constant (z­independent) operators in V, then Y m is exactly a translation map in the

sense of commutative algebra, and we can regard Vasa graded commuta­

tive algebra with operation m-l'

3. We want to take the third perspective: for each v in V, Y (v, z) is a

"quantum operator". We want to ask several questions about these quan­

tum operators:

(i) What sort of properties does Y(v, z) have as a quantum operator on

its own? In other words, what class of quantum operators arises from the

study of VOA's.

(ii) What properties does the space of all vertex operators have as linear

subspace of the space of all quantum operators?

We recall that QO = QO(V) denotes the space of all quantum operators

built from Lin(V). We write QOy = QO(V, Y) as the space of all quantum

operators of the form Y(v, z), v in V, for a fixed VOA structure Y on V. Again we want to characterize in simple terms what sort of subspaces

of QO can be of the form QOy for some VOA structure Y on V. Even­

tually, we want to understand how to concretely construct examples of

VOA's by first constructing an appropriate subspace of QO(V). We need an appropriate generalization of the notion of a creative operator algebra.

Recall that we have defined a sequence of bilinear operations On in QO. It is a consequence of the Cauchy-Jacobi identity that the following holds:

Some Classical and Quantum Algebras 521

Proposition 5.3 Suppose (V,l,Y) is a VOA, and v,v' are in V. Then

for any integer n,

Y(v,z) On Y(v',z) = Y(v(n)v',z).

In particular, QOy is closed with respect to all of the operations On. More­

over for v, v' in V, Y(v,z) On Y(v',z) = 0 (5.19)

for all sufficiently large positive n. This implies that QOy is a local quantum operator algebra (See Definition 3.6). In particular, a vertex operator is

mutually local with itself. This is a highly nontrivial property for a quantum

operator. Motivated by property 2(iii) of YOAs, we introduce the following:

Definition 5.4 A commutative quantum operator algebra 0 is a local QOA

such that for any a(z), b(z) in 0,

a(z)b(w) '"" (-I)lallblb(w)a(z)

in the sense that the matrix entries for both sides represent the same rational function.

Let's now introduce a distinguished element, 1 in the doubly graded

space V.

Definition 5.5 (cf. Definition 5.1) A creative QOA for (V, 1) is a com­

mutative QOA, 0, such that (i) For every a(z) in 0, a(z)1 is a series with only nonnegative powers of

z. (ii) The map 0 -4 V,

is a linear isomorphism. This map is called the creative map associated with O.

The reader should verify that every YOA, (V, 1, Y), defines a creative QOA, QOy. We now give the converse, which is the main new result of this paper:

Theorem 5.6 Let (V,I) be a based space and let 0 be a creative QOA

for (V, 1). Then (V, 1, Y) is a VOA where the vertex map Y is given by the inverse of the creative map associated to O.

522 B. H. Lian and G. J. Zuckerman

This theorem reformulates the Cauchy-Jacobi axiom in VOA theory in

terms of locality, commutativity and creativity in QOA theory. The

reader should compare our approach to that theorem of Frenkel-Lepowsky­

Meurman which reformulates the Cauchy-Jacobi axiom in terms of ratio­

nality, associativity and commutativity [5].

Question: Given a commutative QOA, 0, can we construct a creative QOA?

Proposition 5.7 Let 0 be a commutative QOA on the space V. Then

for each nonzero vector 1 of degrees 111 = 0,11111 = 0, there is a canonical

subquotient 0 1 of 0 which is a creative QOA for some based space.

To construct 01, we proceed as follows: let 0' be the subspace of quantum

operators a(z) in 0 such that a(z)1 has only nonnegative powers of z. Let

V' be the image of the map X : 0' --+ V, a(z) ~ limz-+oa(z)1. We show

that 0' is a closed with respective to all of the operations On, hence is a

commutative QOA on V. In fact by direct computation, we have for any

a( z), b( z) in 0', and for any integer n:

limz-+oa(z) On b(z)1 = a(n)b(-l)1. (5.20)

This implies that the algebra 0' acts in V' by restriction. This means that

0' is now a commutative QOA for the based space (V', 1). Now let I be the two sided ideal generated by ker X in 0'. Consider

the subspace x(I) in V'. If a(z) is in 0' and b(z) in I, then a(z) On b(z) is

in I. It follows from eqn (5.20) that a(n)b(-l)1 is in XCI) for all n. Thus

we have shown that the action of 0' in V' stabilizes the subspace XCI). It

is now clear that 0' acts in V' /x(I) in a natural way. Moreover, we have

an induced isomorphism X : 01 = 0'/1 --+ V' /X(I). This map defines a

creative map associated to the commutative QOA 01 for the based space

(V' /X(I), 1).

5.1 Examples

Let C be the Clifford algebra with the generators ben), c(n) (n E Z ) and

the relations [8]

b(n)c(m) + c(m)b(n)

b(n)b(m) + b(m)b(n)

15n ,-m-l

o

Some Classical and Quantum Algebras 523

c(n)c(m) + c(m)c(n) = 0 (5.21)

Let oX be a fixed integer. The algebra C becomes Z -bigraded if we define

the degrees !b(n)! = -jc(n)! = -1, IIb(n)1I = oX - n - 1, IIc(n)1I = -oX - n. Let /\ * be the graded irreducible C * -module with generator 1 and relations

b(m)1 = c(m)1 = 0, m 2 0 (5.22)

Let b(z), c(z) be the quantum operators

b(z) 2:: b(m)z-m-I m

c(z) 2:: c(m)z-m-I (5.23) m

Let 0 = O(b, c) be the smallest QOA containing b(z),c(z).

Lemma 5.8 The QOA O(b, c) is spanned by the following quantum oper­

ators (see section 3):

with ni > ... > ni 20, mi > ... > mj 2 o.

Let's sketch a proof. If A(z) is in 0, then so is A(z) 0_2 1 = aA(z) (see

eqn (3.10». Thus 0 contains all the derivatives of b(z), c(z). Since 0

is closed with respect to the Wick product, the iterated Wick products of

the derivatives are also in O. So it is enough to show that those iterated

Wick products are closed under every On, ie. if A(z), B(z) are two such

products then A(z) On B(z) is a sum of those products. This can be shown

by induction and by computing the operator product expansions in two

ways: A(z)B(w) and B(w)A(z).

By calculating the operator product expansion, one will discover that

in fact 0 is commutative as a QOA. Let's consider the map X : 0 --+ /\,

a(z) 1-+ limz-+oa(z)1. Using the quantum operator given in the above Lemma, we get

limz-+o : an1b(z)··· anib(z) am1c(z)··· amic(z) : 1 = nIl··· nilml1

... mjlb( -ni - 1)··· b( -ni - 1)c( -mi - 1)··· c( -mj - 1)1. (5.24)

Since the vectors on the right hand side form a basis for the space /\, the

map X is necessarily an isomorphism. It follows that 0 is a creative QOA

524 B. H. Lian and G. J. Zuckerman

with the associated creative map x. Thus by Theorem 5.6, (/\ *,1, X-1) is aVOA.

We remarked earlier that the products On in a QOA are in general

neither associative nor commutative as bilinear operations. We can now

give a good example to illustrate this. Let's rescale c(z) by a parameter Ii (Planck's constant) so that we have

b(n)c(m) + c(m)b(n) = M n .-m - 1. (5.25)

It is clear that with this new relation the above construction still goes

through almost word for word, and we obtain a creative QOA Oli. Consider

the following computations:

: (: b(z)c(z) :)b(z) : - : b(z)(: c(z)b(z) :)

: (: b(z)c(z) :)b(z) : - : b(z)(: b(z)c(z) :)

Ii8b(z)

Ii8b(z). (5.26)

The first eqn shows that the Wick product (ie. o-d is nonassociative for

Ii =1= 0. The second eqn shows that it is noncommutative. In fact the

obstruction for associativity and commutativity is of order Ii. In this sense,

Oli is a nonassociative deformation of the commutative associative algebra

0 0 •

We now discuss our second example. Let 9 be a finite dimensional simple Lie algebra with the standard invariant bilinear form B . Let g be

the associated affine Kac-Moody Lie algebra whose bracket is given by

[X(n), Y(m») = [X, Y)(n + m) + n(B(X, Y)6n+m •0 (5.27)

where ( is a nonzero element of the 1 dimensional center of g. This Lie

algebra can be given a Z grading by IX(n)1 = 0, IIX(n)1I = -n for all

XEg.

Let k be a complex parameter and L(g, k) be the graded irreducible

g-module generated by 1, with the relations:

X(n)1 = 0, n ~ 0, X E g, (1 = kl. (5.28)

For each X in g, define the quantum operator

X(z) = L X(n)z-n-l. (5.29) n

This has degrees IXI = 0, IIXII = 1. Let 0 = O(g, k) be the smallest QOA in QO(L(g, k» such that 0 contains all the X(z).

Some Classical and Quantum Algebras 525

Proposition 5.9 O(g,k) is a creative QOA. If {XI ,X2 , ••• } is a basis for

g, then O(g, k) is spanned by

: anu Xl (z)an12 Xl (z) ... an21 X2(z)an22 X 2(z)··· :

The proof is similar to the first example above.

Corollary 5.10 ([7J, see remarks in introduction.) (L(g,k),l,X- I ) is a

VOA, where X is the creative map associated to O(g, k).

6 Main Theorems

We now discuss the main results, which draw from all the notions introduced

in this paper. We will also illustrate the results with some examples which

are built out of the previous ones.

Let 0 be a creative QOA. A quantum operator L(z) is called Virasoro

if for some constant K, (called the central charge of L) [1],

L(z)L(w) i(z - W)-4 + 2L(w)(z - w)-2 + aL(w)(z - w)-l+ : L(z)L(w) :

L(z)a(w) ... + lIalla(w)(z - w)-2 + 8a(w)(z - w)-l+ : L(z)a(w): (6.30)

for all homogeneous a(z) in O. Here " ... " denotes the higher order polar terms.

Definition 6.1 A pair (O,L) is called a conformal QOA if 0 is a creative QOA with a distinguished Virasoro quantum operator L(z).

For example in the QOA O(b, c) introduced above, for a fixed .\, we

have a Virasoro quantum operator [8]

LbC(z) = (.\ - 1) : c(z)8b(z) + .\ : 8c(z) b(z) : (6.31)

with central charge K, = -12.\2 + 12,\ - 2.

In the second example O(g, k), if k + h v i= 0 we define (see [13])

1 LO(z) = 2(k + hV) ~ : Xi(Z)Xi(z) : (6.32)

• where h v is the dual Coxeter number of g and the Xi is an orthonormal

basis relative to the form B. It is a fundamental fact the LO(z) is a Virasoro quantum operator.

526 B. H. Lian and G. J. Zuckerman

6.1 The BRST construction

It is evident that if (0, L), (0', L') are conformal QOAs on the respective

based spaces (V, 1), (V', 1') with central charges "', ",', then (O®O', L+L') is a conformal QOA on (V ® V', 1 ® 1) with central charge", + ",'. From now on we fix .x = 2 which means that Lbc now has central charge -26. Let

(0, L) be any conformal QOA with central charge", and consider

C*(O) = O*(b, c) ® O. (6.33)

This QOA has a natural quantum operator called the BRST current [8]

[3][4]:

J(z) =: c(z)(L(z) + !Lbc(z» : .

which has degrees IJI = IIJII = 1.

Lemma 6.2 [14][3][4] Let Q = ReszJ(z). Then Q2 = 0 iff '" = 26.

(6.34)

Recall that for any element a(z) in the QOA C*(O), we have J(z)ooa(z) =

[Q, a(z)]. Thus the graded commutator [Q, -] is a derivation of the QOA

C*(O). For", = 26, [Q, -] is a differential on the QOA C*(O) and we have

a cochain complex

[Q, -] : C*(O) ~ C*+l(O). (6.35)

It is called the BRST complex associated to O. Its cohomology will be

denoted as H*(O). All the operations On on C*(O) descend to the coho­

mology. However, all but one is trivial.

Theorem 6.3 [24][26][19] The Wick product 0_1 induces a graded com­

mutative product on H*(O) with unit element represented by the identity

operator. Moreover, every cohomology class is represented by a quantum

operator a(z) with lIall = o.

Consider now the linear operator Ll : C*(O) ~ C*-l(O), a(z) 1--+

b(z) 01 a(z).

Theorem 6.4 [19] The operator Ll descends to the cohomology H*(O). Morever, it is a BV operator on the commutative algebra H*(O). Thus

H*(O) is naturally a BV algebra.

Some Classical and Quantum Algebras 527

The two main theorems above were originally proved in [19] in the lan­guage of vertex operator algebras. It is Theorem 5.6 which allows us to

translate between the two versions. (For related versions of Theorem 6.4,

see [11][22][15][12].) Examples: Recall that O(g, k) is a conformal QOA with a Virasoro quantum

operator L9(Z). Its central charge is '" = \~,!:v9. Since the dimension of a simple Lie algebra g is never 26, we can choose k so that '" = 26. With this choice, we can consider the BRST cohomology H* = H*(O(g, k». The

answer is given as follows:

HO 9!': C

Hl '" g

H2 9!': g

H3 C,{ C. (6.36)

Recall that given a Lie algebra, its exterior algebra is a BV algebra. As a

BV algebra, the structure of H* is determined by the following:

Proposition 6.5 There is a unique surjective homomorphism of BV alge­

bras A* g ~ H*(O(g,k» such that under the above identification (6.36),

X t--+ X E Hl, for all X in g.

References

[1) A. Belavin, A.M. Polyakov and A.A. Zamolodchikov, "Infinite confor­

mal symmetry in two dimensional quantum field theory", Nucl. Phys.

B241 (1984) 333.

(2) R.E. Borcherds, "Vertex operator algebras, Kac-Moody algebras and

the Monster", Proc. Natl. Acad. Sci. USA. 83 (1986) 3068.

(3) B. Feigin, "Semi-infinite homology for Virasoro and Kac-Moody alge­bras", Usp. Mat. Nauk. 39 (1984) 195-196.

[4] I.B. Frenkel, H. Garland and G.J. Zuckerman, "Semi-infinite cohomol­ogy and string theory", Proc. Nat. Acad. Sci. U.S.A. 83 (1986) 8442.

[5] I.B. Frenkel, J. Lepowsky and A. Meurman, "Vertex Operator Alge­bras and the Monster", Academic Press, New York, 1988.

528 B. H. Lian and G. J. Zuckerman

[6] LB. Frenkel, Y. Huang and J. Lepowsky, "On axiomatic approaches to

vertex operator algebras and modules", Yale-Rutgers preprint, Dept. of Math., 1989.

[7] LB. Frenkel and Y. Zhu, "Vertex Operator Algebras associated to

affine Lie algebras and the Virasoro algebra", Duke Math J. 66 (1992)

123.

[8] D. Friedan, E. Martinec and S. Shenker, "Conformal invariance, su­

persymmetry and string theory", Nucl. Phys. B271 (1986) 93.

[9] M. Gerstenhaber, "The cohomology structure of an associative ring",

Ann. of Math. 78, No.2 (1962) 267.

[10] M. Gerstenhaber, "On the deformation of rings and algebras", Ann.

of Math. 79, No.1 (1964) 59.

[11] E. Getzler, "Batalin-Vilkovisky algebras and two dimensional topolog­

ical field theory", MIT preprint, hepth/9212043.

[12] Y-Z. Huang, "Operadic formulation of topological vertex algebras and

Gerstenhaber or Batalin-Vilkovisky algebras", Univ of Penn preprint,

hepth/9306021.

[13] V. Kac and A. Raina, Highest weight representations of infinite dimen­

sional Lie algebras, Advanced Series in Math Physics, Vol. 2, World

Scientific (1987).

[14] M. Kato and K. Ogawa, "Covariant quantization of string based on

BRS invariance", Nucl. Phys. B212 (1983) 443.

[15] T. Kimura, J. Stasheff and A. Voronov, "On operad structures of mod­

uli spaces and string theory", UNC preprint, hepth/9307114.

[16] B.H. Lian and G.J. Zuckerman, "New selection rules and physical

states in 2d gravity; conformal gauge", Phys. Lett B254, No.3,4 (1991)

417.

[17] B.H. Lian and G.J. Zuckerman, "2d gravity with c=l matter" ,Phys.

Lett. B266 (1991) 21.

Some Classical and Quantum Algebras 529

[18] B.H. Lian and G.J. Zuckerman, "Semi-infinite homology and 2D grav­

ity (I)", Commun. Math. Phys. 145 (1992) 561.

[19] B.H. Lian and G.J. Zuckerman, "New perspectives on the BRST­algebraic structure in string theory" , hepth/9211072, Commun. Math. Phys. 154 (1993) 613.

[20] G. Moore, "Finite in All Directions", Yale preprint, hepth/9305139.

[21J M. Penkava, "A note on BV algebras", UC Davis preprint November 92.

[22] M. Penkava and A. Schwarz, "On some algebraic structures arising in string theory", UC Davis preprint, hepth/9212072.

[23] A. Schwarz, "Geometry of Batalin-Vilkovisky quantization", UC Davis preprint, hepth/9205088.

[24] E. Witten, "Ground ring of the two dimensional string theory",

hepth/9108004, Nucl. Phys. B373 (1992) 187.

[25] E. Witten, "A note on the anti-bracket formalism", preprint IASSNS­

HEP-90/9.

[26] E. Witten and B. Zwiebach, "Algebraic structures and differential ge­ometry in 2d string theory", hepth/9201056, Nucl. Phys. B377 (1992) 55.

Department of Mathematics, Harvard University, Cambridge, MA 02138

Department of Mathematics, Yale University, New Haven, CT 06520

Received March 14, 1994

Total Positivity in Reductive Groups

G. Lusztig*

Dedicated to Bert Kostant on his 65th birthday

An invertible n x n matrix with real entries is said to be totally ~ 0 (resp. totally> 0) if all its minors are ~ 0 (resp. > 0). This definition appears in Schoenberg's 1930 paper [Sj and in the 1935 note [GKj of Gantmacher and Krein. (For a recent survey of totally positive matrices, see [Aj.)

In [Wj, Ann Whitney proved a reduction theorem for totally ~ 0 ma­trices, based on a lemma of Fekete [FPj. As Loewner noted in [Loj, Whit­ney's theorem has the following consequence: the set of n x n invertible totally ~ 0 matrices is precisely the submonoid of GLn(R) generated by diagonal matrices with > 0 diagonal entries and by matrices with 1 on diagonal and with a single non-zero off-diagonal entry which is > 0 and is just above or just below the diagonal.

In this paper, the monoids G?o and G>o, are defined for an arbitrary split reductive connected algebraic group Gover R. We define G?o as the submonoid of G generated by a certain set of elements (see 2.2) analogous to those which enter in Whitney's theorem for GLn(R); we define G>o as a certain submonoid (without 1) of G?o (see 2.12). The definition of G?.o (resp. G>o) reduces for G = GLn(R) to the classical definition of totally ~ 0 (resp. totally> 0) matrices, by Whitney's theorem (resp. by the proof of Whitney's theorem).

One of the main tools in our study of G>o and G>o is the theory of canonical bases in [L1j. Thus, our proof of the fact that G>o is closed in G (Theorem 4.3) is based on the positivity properties of the canonical bases (in the simply-laced case), proved in [L1j,[L2j, which is a non­elementary statement, depending ultimately on the Weil conjectures. The same holds for our proof of Theorem 5.6 which asserts that all elements in G>o are regular, semisimple and R-split. (For G = GLn(R), this theorem is due to Gantmacher and Krein [GK,KGj.) Since the positivity properties of canonical bases are available only in the simply-laced case, we systematically prove results in the simply-laced case first, and then treat the general case by "descent".

Some of our results seem to be new even for GLn(R). This is the case for example in the study of unipotent elements of G?o (see §6) and in the study of a certain remarkable polyhedral subspace B>o of the real flag manifold (see §8). The same holds for our descriptio~ (see 4.8) of G>o as a connected component of the intersection of the big Bruhat double

*Supported in part by the National Science Foundation.

532 G. Lusztig

coset with respect to a certain Borel subgroup and the big Bruhat double coset with respect to an opposite Borel subgroup.

The definition of G~o makes sense also in the case where real numbers are replaced by the field of real rational functions with a suitable notion of positivity (see 2.1{d». In this case, G>o has a natural partition into "zones" (see §9). The combinatorics which enters in the parametrization of these zones is remarkably similar to the combinatorics which enters in the definition of canonical bases attached to the Langlands dual of G. A precise statement in this direction is given in §1O. (This statement appeared in [L2, 42.2.8], in the simply-laced case, but it is much more striking in the non-simply laced case, where the Langlands dual enters.)

I was introduced to the classical totally positive theory for GLn by Bert Kostant, who pointed out to me its beauty and asked me about the possible connection with the positivity properties of the canonical bases of [Ll]. It is a great pleasure to thank him here.

Contents

1. Pinnings. 2. The monoid G~o. 3. G-modules. 4. Over real numbers: G~o is closed in G. 5. Over real numbers: regularity theorem. 6. Over real numbers: unipotent elements in G?o. 7. Over real numbers: curves in 0-. 8. Over real numbers: the subsets B>o, B>o of the flag manifold. 9. Real rational functions and the zones ~f uto and of G?o. 10. A relationship between zones and canonic-al bases. 11. The elements uw,p E U+.

1. Pinnings

1.1. Let K be an infinite field. We shall often identify an algebraic variety defined over K with the set of its K-rational points. Let G be a split reductive connected algebraic group over K. We assume that we are given a K -split maximal torus T of G and a pair B+, B- of opposed Borel subgroups containing T, with unipotent radicals U+, U-. Let Ui+ (i E J) be the simple root subgroups of U+ and let Ui- (i E J) be the corresponding root subgroups of U-. Here J is a (finite) indexing set. Let X~ : T -+ K* be the (simple) root corresponding to ut (then the root X~-l corresponds to Ui-). Let Xi : K* -+ T be the (simple) corrot corresponding to X~. We assume that for any i E J we are given

Total Positivity in Reductive Groups 533

isomorphisms Xi : K .:::. ut and Yi : K .:::. Ui-, as algebraic groups over K, such that the assignment

(where a, e E K, bE K*) defines a homomorphism SL2(K) -+ G. The da­tum (T, B+, B-, Xi, Vi; i E I) as above is said to be a pinning (in French: epinglage) for G. It is known that we can always find a pinning for G and that any two pinnings for G are conjugate under G.

Let Si E G be the image of ( ~ 1 ~) under the homomorphism ( a).

Note that Si belongs to the normalizer of T.

1.2. There is a unique isomorphism W : G .:::. GOpp (the opposite group structure) such that w(xi(a» = Yi(a), w(Yi(a» = xi(a) for all i E I,a E K and wet) = t for all t E T.

1.3. The following identities hold: (a) tXi(a) = Xi(x~(t)a)t, tYi(a) = Yi(xW)-la)t for i E I, t E T, a E

K; (b) Yit(a)xi2(e) = xi2(e)Yit(a) for a,e E K and il =I- i2 in I.

Moreover, for i E I we have (c) Yi(a)Xi(b-1)xi(e) = xi(d)Xi(b')Yi(a')

where a' = a/Cae + b2), b' = b/(ae + b2), d = e/(ae + b2)j a = a' /(a'e' + b,2), b = b' /(a'd + b,2), e = d /(a'c' + b,2).

(Here we assume that a, b, e E K, b =I- 0, ac + b2 =I- 0 or equivalently that a', b', e' E K,b' =I- O,a'd +b,2 =I- 0.)

1.4. We shall denote by Y (resp. X) the (free abelian) group of all homomorphisms of algebraic groups K* -+ T (resp. T -+ K*). Let (,) : Y x X -+ Z be the pairing defined by x(y(a» = a(Y'x} for all aE K*. G is said to be simply laced if for any it =I- i2 in I we have (XiII X~2) = {X i2 , xU· In this case, the Coxeter graph of G has by definition I as set of vertices and it =I- i2 form an edge precisely when (Xit' X~2) = -1.

1.5. Let G be a semisimple, simply connected algebraic group defined and split over K, with a pinning (1', iJ+, iJ-, Xj, Yj;i E J) (over K) and with a given automorphism 0' : G -+ G (over K) such that conditions (a)-(d) below are satisfied.

( a) G is simply laced. (b) 0' preserves the pinning, that is, it leaves stable 1', iJ+ and iJ­

and, for a suitable permutation i -+ O'(j) of J, we have O'(xj(a» = Xu(j) (a), O'(Yj(a» = Yu(j) (a) for all i E J, a E K.

534 G. Lusztig

(c) If jl f:. j2 are in the same orbit of q : J -+ J, then jbh do not form an edge of the Coxeter graph.

(d) j and q(j) are in the same connected component of the Coxeter graph, for any j E J. The definitions in 1.1 are applicable to a instead of G. In particular, Xj : K" -+ T, xj : T -+ K" denote the simple coroots and simple roots of a; for j E J we have a well defined element Sj in the nor­malizer of T in a. Let U+ (resp.U-) be the unipotent radical of B+ (resp. B-). Let J be the set of orbits of q : J -+ J and let au, TCT, B+ CT , B-CT be the fixed point sets of q on a, T, B+, B-. Then aCT is a semisimple, simply connected algebraic group defined and split over K with a pinning (TCT,B+u,B-U,xp,Yp;p E J). Here xp(a) = njEpxj(a), Yp(a) = njEP Yj(a) for p E J, a E K. (The order in the products is immaterial, by (c).)

1.6. We return to the setup in 1.1. Let G be the (group of K-rational points of the) simply connected algebraic covering of the derived group of G and let 71" : G -+ G be the covering homomorphism. The pinning for G induces a pinning (T,jj+,jj-,Xi,iJi;i E J) for G. Here T = 7I"- I T,jj+ = 71"-1 B+, jj- = 71"-1 B-. Moreover, 7I"(xi(a» = xi(a),7I"(Yi(a» = Yi(a) for i E J, a E K. Let [;+, [;- be the unipotent radicals of jj+, jj-. It is known that we can find (in an essentially unique way) a with a pinning (T,B+,B-,xj,Yj;j E J) and q: a -+ a as in 1.5, and an isomorphism </> : aCT ~ G (as algebraic groups over K) which is compatible with the pinnings on aCT, G.

In the rest of the paper G, G, a, their pinnings and q, </> are fixed as above.

2. The monoid G?o

2.1. We now assume that we are given a subset K>o of K -{O} containing 1, such that the following holds:

(a) if f,J' E K>o, then f + f' E K>o, ff' E K>o, 1/ f E K>o. Then K is necessarily of characteristic zero. We set K>o = K>o U {O}.

For example, we could take (b) K = R,K>o = {x E Rlx > O}, (c) K = R( (f» where f is an indeterminate and K >0 is the subset of

K consisting of all power series of the form f = asfs + as+lfs+1 + ... such that s E Z and as > 0 (we then set Ifl = s.)

(d) K = R(f), the subfield of R((f» consisting of rational functions; the set K>o is obtained by intersecting the analogous set for R((f») (see (c» with K. For f E K, we define If I as in (c).

Total Positivity in Reductive Groups 535

2.2. Let uto be the submonoid (with 1) of U+ generated by the elements xi(a) for ~rious i E I,a E K?o. Let U;o be the submonoid (with 1) of u- generated by the elements Yi(a) for-various i E I,a E K?o. Let T>o be the submonoid (with 1) ofT generated by the elements x(a) for various X E Y,a E K>o. Let G>o be the submonoid (with 1) of G generated by the elements xi(a),Yi(a) for various i E I,a E K>o and by the elements x(a) for various X E Y,a E K>o. Clearly, we have

w(uto) = U;o' w(U;o) = uto' W(T>o) = T>o, W(G?o) = G?o. The definiti~s above ~an be given in the same way when G, U+, ...

- -+ .. + --+ are replaced by G, U ,... or by G, U ,.... Hence G?o, U>o' . .. and . .+ -

G?o, U?o, . .. are defined.

Lemma 2.3. (a) If u+ E uto' t E T >0, u- E U;o, then u+tu- E G?o and u-tu+ E G?o. - -

(b) Any element 9 E G>o can be written uniquely in the form 9 = u+tu- with u+ E U~o' t E T>o, u- E U:;'o.

(c) Any element 9 E G>o can be written uniquely in the form 9 = u1ltut withut E U~o,tl ET>o,ul E U:;'o.

(a) is obvious; the existence part in (b) and (c) follows by repeated applications of 1.3(a),(b),(c). (Note that the condition ac + b2 :/= 0 in 1.3(c) is automatically satisfied for a,c E K>o,b E K>o since in fact we have ac + b2 E K>o.) The uniqueness part in (b) and (c) follows from properties of the Bruhat decomposition.

2.4. Let W = normalizer(T)IT be the Weyl group of G. For i E I, let Si be the image of Si in W. Similarly, let W = normalizer(T) IT be the Weyl group of G; for j E J, let Sj be the image of Sj in W. Recall that W is a Coxeter group on generators {Si Ii E I} and W is a Coxeter group on the generators {SjJj E J}. Let 1 : W ~ Nand i: W ~ N be the standard length functions. Let Wo E Wand Wo E W be the elements of maximal length. Let u : W ~ W be the automorphism defined by u(Sj) = su(j)

for all j E J. We may identify W with the fixed point set Wu. If i E I, we may regard i as a u-orbit in J and we then have Si = njEi Sj. With this identification we have Wo = Woo

Proposition 2.5. Let i :/= if in I. (a) If SiSi' = Si'Si, then Xi (a)Xil (b) = xdb)xi(a) for all a,b E K. (b) If SiSi' Si = Si' SiSi', and a, b, c E K satisfy a + c :/= 0, we have

536

(a) is obvious and (b) follows from the identity

G~nG!nGrn ~ G ! ;)(~ r n G ! D

in SL3(K), where a' = be/(a + c), b' = a + c, c' = ab/(a + c).

We shall prove the following two results simultaneously.

G. Lusztig

Proposition 2.6. Let i ¥- i' be elements of I and let m ~ 2 be the order of SiSi' in W. Let at, a2, ... ,am be a sequence in K >0. There is a unique sequence a~, a~, ... ,a~ in K >0 such that

where both products consist of m factors.

Proposition 2.7. Let wE W. Let i l ,i2 , ... ,in be a sequence in I such that Sil Si2 ... Sin is a reduced expression for w.

(a) The map K~o - U+ given by

is injective. (b) The image of the map (a) is a subset U+(w) of U;o which does

not depend on it, i2 , ••• , in. -(c) If w' E W is distinct from w, then U+(w) n U+(w') = 0.

First note that, if a E K - {a}, then xi(a) E B-siB-. Hence, if it, i 2 , ... , in is as in (a) and aI, a2,"" an are non-zero elements of K, we have

(d) Xil(at}Xi2(a2) ... Xin(an) E B-SiI B-Si2 B- ",SinB-C B-Sil Si2 ••. Sin B-

(by properties of the Bruhat decomposition). To prove (a), it suffices to verify the following statement:

if Xil (al)xi2 (a2) ... Xi" (an) = XiI (aDXi2 (a~) ... Xi" (a~) where al,· .. ,an and ai, ... ,a~ are in K - {a}, then al = a~ for all l.

We argue by induction on n. Our assumption implies

Total Positivity in Reductive Groups 537

Assume that at - a~ # o. If at - a~ # 0, the two sides of the last equality are in B-Si1 Si2 ... SinB- and B-Si2Sia ... SinB- (by (d)). This is in contradiction with the Bruhat decomposition. Thus, we must have at = a~. Hence

xi2(a2) ... Xin(an) = xi2(a~) ... Xin(a~)

and the induction hypothesis shows that a2 = a~, ... , an = a~. Thus, (a) is proved.

Next we show that (b) holds whenever Proposition 2.6 holds. Let i~, i~, ... , i~ be another sequence as in (a). By a theorem of Iwahori and Tits, we can pass from it, i2 , ••• , in to i~, i~, ... , i~ by a chain of elementary steps corresponding to braid group relations. Hence we may assume that i~, i~, ... ,i~ is obtained from it, i2, ... ,in by replacing a string of m consecutive terms i, i', i, ... by the m terms i', i, i', .... (Here i # i' and m is the order of SiSi' in W.) Then the desired conclusion follows immediately from Proposition 2.6. Now if G is simply laced, then in the argument above we always have m = 2 or 3, and in this case Proposition 2.6 holds by 2.5. (We use that the condition a + c # 0 in 2.5(b) is automatically satisfied if a E K>o,c E K>o, since in fact a + c E K>o.) Hence for simply laced G, (b) holds unconditionally. In particular, (b) holds for G instead of G. We now prove Proposition 2.6. In the setup of Proposition 2.6, we consider the element

(e) TIjEi xj(at) TIjEi' Xj(a2) TIjEi Xj(a3) TIjEi' Xj(a4) ...

of U+ (the number of TI signs is m). Applying (b) (in G) we see that this element is equal to

(f) TIjEi' xj(a~,j) TIjEi xj(a~,;) TIjEi' xj(a;,j) TIjEi xj(a~,j) ... (again the number of TI signs is m). Here a~,j' a~,j' . .. are elements in K>o. (We use the fact that the elements

and

II Sj II Sj II Sj II Sj ... jEi' jEi jEi' jEi

of Ware equal and have length equal to the number of Sj used.) Since the element (e) is fixed by a, the same is true about the element (f). Writing the fact that (f) is fixed by a and using the injectivity in (a), we deduce that a~,j is independent of j (we denote it by a~), a~,j is independent of j (we denote it by a~) and so on. The equality of (e) and (f) can then be rewritten in the form

538 G. Lusztig

and Proposition 2.6 is proved. By the previous argument we now see that (b) holds unconditionally. By (d), we have U+(w) C B-SiISi2 ",si"B-, so that (c) follows from Bruhat decomposition. This completes the proof of Proposition 2.7.

Corollary 2.8. We have U~o = UwEwU+(w).

Let U' = UwEwU+(w) (a disjoint union, by 2.7(c». By definition, we have U' C uto. It is enough to show that, for any i E I, a E K>o, we have xi(a)U'C U'. We show that, for w E W:

(a) if I(SiW) > l(w), we have xi(a)U+(w) C U+(SiW);

(b) if l(siw) < l(w), we have xi(a)U+(w) C U+(w). Now (a) is clear from the definitions. In the setup of (b) we have U+ (w) = UbXi(b)U+(SiW) (where b runs over K~o) hence

and (b) is proved. It follows that xi(a)U' C U'. The corollary is proved.

2.9. Entirely analogous results hold for U?o instead of U~o' Namely, given wE W, and i},i2, ... ,in as in 2.7, the map K::;'o -+ U- given by (al,a2, ... ,an) ~ Yil(al)Yi2(a2) ... Yi,,(an) is injective; its image is a subset U-(w) of U:;o which does not depend on ito i2, ... , in. We have U:;o = UWEWU-(w). Note that \lI(U+(w» = U-(w- 1), \lI(U-(w» =

U+(w- 1).

The following result will be needed in §4.

Lemma 2.10. Let i = (i}, i2,"" iN) be a sequence in I such that Sil Si2 ... SiN is a reduced expression for woo Then the image of the map K'1o -+ U+ given by (al,a2, ... ,aN) -+ xiJ(at}xi2(a2),,,xiN(aN) is equaf to U~o'

Let Uj+ be the image of this map. We show that

(a) U1+ is independent of i.

It suffices to show that ut = u1t, whenever i' = (ii, i~, ... , i'N) is obtained from i by replacing a string of m consecutive terms i, i', i, ... by the m consecutive terms i', i, i', ... where m is the order of SiSi' in W. We are then reduced to the case where I = {i, i'} and i = (i, i', i, ... ), i' = (i',i,i', ... ). Let 9 E ut. We have 9 = xi(at}xe(a2)xi(a3) ... (m factors) where a},a2,'" ,am are in K~o. If al == 0, then 9 = Xi'(a2)xi(a3) .. ' (m - 1 factors) and inserting at the end a factor Xi(O) (if m is even) or Xi' (0) (if m is odd), we see that 9 E Ui;. If am = 0, then 9 = xi(al)xi,(a2)xi(a3) ... (m-1 factors) and inserting in the front a factor xdO) we see that g E u1t. If as = 0 for some S E [1, m - 1],

Total Positivity in Reductive Groups 539

then the s-th factor can be suppressed in 9 and the (s - 1) and (s + 1) factor combine to a single factor; inserting in the front a factor Xi' (0) and at the end a factor Xi(O) (if m is even) or Xi'(O) (if m is odd), we see that 9 E Ui1". Finally, if at. a2, a3, ... are all in K>o, then 9 E Ui;- by Proposition 2.6. Thus, ut c Ui;-. Similarly, we have Ui;- c ut hence Uj;- = Ui+. Thus, (a) is proved. Clearly, xi(a)Ut C ut if i starts with i and a E K>o. Since any i' E I is the first term of some i as above, and ut is independent of i, we deduce that xda)ut C ut for any i' E I. It follows immediately that ut = U1o. The lemma is proved.

2.11. For w,w' E W, let G(w,w') = U-(w)T>oU+(w'). Using repeated­ly 1.3(a),(b),(c), we see that we have also G(w,w') = U+(w')T>oU-(w). From 2.8, 2.9, we see that G~o = Uw,w'EWG( w, w'). It is clear that \lI(G(w,w')) = G(w,-1,w-1).

2.12. The monoid G>o. We define

U:;o = U+(wo), U:;o = U-(wo),

G>o = G(wo,wo) = U:;oT>oU:;o = U:;oT>oU:;o·

Let i E I,a E K~o. Since l(siwo) < l(wo), we have xi(a)U:;O C U:;o (for a E K>o, this follows from 2.8(b); for a = 0, this is trivial.) Applying this repeatedly, we see that U1oU:;o c U:;o. Similarly, we have

We have

and U?'oG>o C U?oU:;oT>oU:;o c U:;oT>oU:;o = G>o.

It follows that G~oG>o C G>o. Similarly, we have G>oG~o C G>o. In particular, G>oG>o C G>o, so that G>o is a submonoid (without 1) of G~o·

Proposition 2.13. Let n+ = B+mB+, n- = B-mB-, where m is an element in the normalizer ofT representing wo0 We have

(a) U:;o = U10 n n-, (b) U:;o = U?o n n+,

(e) G>o = G~onn+ nn-.

By 2.8, we have U:;o n n- = UwEWU+(w) n n-. By the proof of 2.7, U+(w) is contained in B-mwB- where mw represents w in the normalizer of T. Hence U+ (w) n n- is empty, if w =f. Wo, and is equal to

540 G. Lusztig

U+(Wo) = U~o, if W = Woo This proves (a). The proof of (b) is entirely similar.

We prove (c). Using (a),(b), we have

and

G>o = U~oT>oU:;o c U~oT>on+ c n+,

hence G>o C G>o n n+ n n-. Conversely, let 9 E G>o n n+ n n-. Using 2.3(b), we can ;-rite uniquely 9 = u+tu- where u+ E U~o' t E T>o, u- E

U:;o. Since u+tu- E n- and tu- E B-, we have u+ -E n-j hence by

(a), we have u+ E U~o. Since u+tu- E n+ and u+t E B+, we have u- E n+, hence by (b), we have u- E U:;o. It follows that 9 E G>o. The proposition follows.

Lemma 2.14. (a) Given WI, W2 E W, there is a unique W3 E W such that U+(WI)U+(W2) = U+(W3).

(b) The assignment WI, W2 1-+ WI *W2 := W3 defines a monoid structure onW.

We prove (a). Let Sil ... Sin be a reduced expression for WI. Clearly, U+(WI) = U+(Sil) ... U+(Sin). Hence to prove (a), we may assume that WI = Si for some i. From the definitions, U+(Si)U+(W2) is equal to U+(SiW2), if l(SiW2) = l(W2) + 1, and to U+(W2), if l(SiW2) = l(W2) - 1. Thus, (a) is proved. (b) follows immediately from (a).

Lemma 2.15. Let Wt, wi, W2, w~ be elements of W such that WI :$ W2 and wi :$ w~ (in the standard partial order of W). Then WI *W2 :$ wi *W~.

We may assume that either WI = W2 or wi = w~. The two cases being similar, we assume for example that WI = W2. By writing WI as a product of l(WI) simple reflections, we are reduced to the case where W = Si for some i E I (and wi :$ w~).

If l(Siwi) = l(wi) + 1, l(SiW~) = l(w~) + 1, we have siwi :$ SiW~ hence Si *wi :$ Si *w~.

If l(Siwi) = l(wi) - l,l(siw~) = l(w~) - 1, we have wi :$ w~ hence Si * wi :$ Si * w~.

If l(Siwi) = l(wi) -1,l(SiW~) = l(w~) + 1, we have wi :$ SiW~ hence Si *wi :$ Si *w~.

If l(SiWi) = l(wi) + l,l(siw~) = l(w~) -1, we have siwi :$ w~ hence Si *wi :$ Si *w~.

The lemma follows.

Total Positivity in Reductive Groups 541

Lemma 2.16. For W E W we define the support of W to be the set of all i E I such that Si appears in some (or any) reduced expressions ofw. Let Si l Si2 • • • SiN be a reduced expression of Wo in W. Let WI. W2, ••• , WN be elements of W such that for any m E [1, Nj, the support of Wm contains i m • Then WI *W2 * ... *WN = Woo

From our assumption, we have Sim :$ Wm for m E [1, Nj. Applying several times the previous lemma, we obtain Sil * Si2 * ... * SiN :$ WI * W2 * ... * WN· Thus, Wo :$ WI * W2 * ... * WN. The lemma follows.

H H' 2.17. The subsets G>o • Let r : U+ ~ KI be the unique homomor-phism of algebraic grou-ps such that r(xi(a)) = a8i for all i E I, a E K, where 8i : I ~ K is the delta function at i. This restricts to a homomor­phism of monoids r' : U;;o ~ K~o. We have a partition K~o = UHKSfo, where H runs over the subsets of I. (we identify a function H ~ K>o with a function I ~ K;::::o which is zero on I - H). Let u;t = {g E

U;;olr'(g) E KSfo}. Clearly, U;oH is the union of all U+(w) with W of support H (see 2.16). -

It is clear that we have a partition (a) U~o = UHU~oH. Similarly, we have a partition (b) U~o = UHU~oH

where U:;OH is the union of all U-(w) with W of support H (see 2.16). Using 2.3, it follows that we have a partition

HH' (c) G;::::o = UH,H'G;::::o

where H, H' run over the various subsets of I and

(d) GH,H' U+,H T U-,H' U-,H'T U+,H ;::::0 = ;::::0 >0;::::0 = 2:0 >0 2:0 •

(The second equality in (d) follows easily from 1.3(c).)

2.18. Oscillatory elements in G;::::o. An element 9 E G;::::o is said to be oscillatory if gm E G>o for some integer m ~ 1. Let G>,o be the set of oscillatory elements of G;::::o. We have -

(a) G>o c G?;o c G;::::o. The definition of G>'o' as well as the following result, were given in

the case of GLn(R) in lGKj,[KGj.

p 't' 2 19 GOsc G1,I roposl Ion . . ;::::0 = ;::::0'

Let 9 E G>,o. By 2.14(c), we have 9 E G!;.OHI for some H, H'. By

definition we ~an find m ~ 1 such that gm E G~o. Since G!;.OH' is closed

under multiplication, we have gm E G!;.OH'. Since G>o C G}b (from the

definition) we have gm E G~b. By 2.14(c), the subsets G~;t, G~b are

542 G. Lusztig

disjoint unless H = I, H' = I. It follows that we have H = H' = I. We have proved that Gosc C CI,1

>0 >0'

We prove the opposite i~clusion G~~ C G~o' Let 9 E G~~. We will show that 9N E G>o (We use the notation of 2~I6.) We write 9 = 9+909-where 9+ E U~r/,90 E T>o, 9- E U~{ Then

N _ ( + + +) 10( - - -) 9 - 9192 ... 9N 9 9192' .. 9N

£, + + +. U+,1 10 T d - - -or some 91 ,92"" ,9N m >0' some 9 E >0 an some 91 ,92"" ,9N in U:;r/. For m E [1, Nj, ~ have 9~ E U+(wm ) where Wm E W has support Ij in particular, the support of Wm contains im (notation of 2.16). By 2.14,2.16, we have

9{ 9i ... 9t E U+(Wt}U+(W2) ... U+(WN) = U+(W1 * W2 * ... * WN)

= U+(wo) = uto.

Similarly, we have 9192 ... 9'N E U:;o. Thus, 9N E UtoT>oU:;o = G>o. The proposition is proved.

2.20. A number of definitions in this section (namely the definitions of

U;o,U±(w),G?o,G(w,w'),G~oH') remain valid in the case where G is replaced by a group associated to a Kac-Moody Lie algebra. The results in 2.7, 2.8, 2.11, 2.14 remain valid. Although G>o is not defined in this generality, we can still define G~o by the equality in 2.19. We will not pursue this generalization any further.

3. a-modules

3.1. Let V be the enveloping algebra of the Lie algebra (over K) of aj this can be defined by generators ej, hj, Ii (j E J) and the usual Serre relations. For any >. = p.j) E N J, there exists a finite dimensional simple V-module A>.. with a non-zero vector 7J>.. such that ej7J>.. = 0 and hj 7J>.. = >'-j7J>.. for all j E J. Moreover, the pair (A>..,7J>..) is determined up to unique isomorphism. There is a unique a-module structure on A>.. such that for any j E J, a E K, we have

(The exponentials are well defined since ej, Ii act nilpotently on A>...) If j E J,a E K*,z E A>..,n E Z satisfy hjz = nz, then Xj(a)z = anz. Let V+ (resp. V-) be the subalgebra of V generated by the elements ej (resp. Ii)j this is the enveloping algebra of a nilpotent Lie algebra. Let B- be the canonical basis of the K-vector space V- obtained by specializing at

Total Positivity in Reductive Groups 543

v = 1 the canonical basis [Ll] of the corresponding quantized enveloping algebra and then extending scalars from Q to K. For A E N J , let B.x be the image in A.x of the set {b E B-lb77.x 1:- O} under the map b 1--+ b77.xj this map is injective and its image B.x is a basis of A.x (see [Ll]). By definition, B.x contains 77.x.

Given a K-linear map A.x - A.x, the matrix of this linear map, its entries and its diagonal entries, are understood to be with respect to the canonical basis B.x.

Proposition 3.2. Let A E NJ and let 9 E G?o.

(a) All entries of the matrix of 9 : A.x - A.x are in K?o.

(b) The product of all diagonal entries of the matrix in (a) belongs to 1 + K?o. In particular, all these diagonal entries are in K>o.

(c) If 9 E G>o, then all entries of the matrix of 9 : A.x - A.x are in K>o·

We prove (a). If (a) is true for 9 and g' (in G>o), then it is also true for their product gg'. Hence to prove (a), we ~ay assume that 9 is one of the generators xj(a},Yj(a} with j E J,a E K?o or Xj(a) with j E J, a E K>o. Now the matrix of Xj(a) : A.x - A.x has diagonal entries of form an for various n E Z and non-diagonal entries equal to OJ moreover, an E K>o for a E K>o, n E Z. Since xj(a} = 1+ E:~l anej In! and Yj(a) = 1+ E:~l an ff In! for some no ::::: 1 (as linear maps A.x - A.x) and an E K>o for a E K>o,n E N, we see that it suffices to observe that for any n E- N, the matrix of ej In! has all entries in N (this follows from [L2, 22.1.7]), and that the matrix of ff In! has all entries in N (this follows from [Ll, 10.11]). This proves (a).

We prove (b). For any g' E G, let Sb(g') be the (b, b)-diagonal entry of the matrix of g' : A>, - A.x. Let us write 9 = g+ gOg- with g+ E u~o' gO E T>o, g- E u:;;o. As in the proof of (a), we see that for any bE B.x, we have Sb(g-) E 1 +-K?o, Sb(g+) E 1 + K?o hence Sb(g) E Sb(gO} + K?o. Taking the product over bE B.x, we obtain Db Sb(g) E Db sb(gO)+K?o. It is now enough to show that Db Sb(gO} = 1. Since the matrix of gO is diagonal, this is equivalent to det(gO, A.x} = 1. This in turn is a consequence of the following more general result: det(g', A>,} = 1 for any g' E G, which follows from the fact that G is generated by unipotent elements.

We prove (c). Fix a sequence jl,j2, ... ,jv in J such that sit sh ... Sj" is a reduced expression of Wo in W. By definition, we have 9 = g+gOg­where

g+ = xit (aDxh (a~) ... Xj" (a~), gO E T>o,

g- = Yit (al}Yh (a2) ... Yj.,(av )

544 G. Lusztig

and ai, a2, ... , av and a~, a~, ... , a~ are in K>o. Let b, b' be two elements of B>.. Let e be the unique element of B>. such that he = 0 for all j. We write

g-(b) - "c-{3 gO({3) = cO/3{3, - ~ /3 ' /3

g(b) = L c/3{3 /3

({3 runs over B>.) and cp,ct,C/3 are in K~o, by (a), cg is in K>o, by (b). It follows that Cb' = ctc~cZ + c where C E K~o. In order to prove that Cb' E K>o, it is therefore sufficient to show that c,;, E K>o and cZ E K>o. Clearly, for some u E U- we have ub = e. By [L3], the monomials fn = fj~l ... fi:" /«nl)!. .. (nv)!) (for various n = (nt, ... , nv) E NV) span U-, hence if we write fnb = E/3 d/3,n{3 (where (3 runs through B>. and d/3,n E K), we have de,n i- 0 for some n. Now g- acts on A>. as '"'" anl an" r h nc - - '"'" nl n" d L.Jn t ... v In, e e c{ - L.Jn at ... av e,n·

Using [L1, 10.11], we have d{,n E K?o for all n. Since d{,n i- 0 for some n, we have de,n E K>o for some n and hence En a~l ... a~"de,n E K>o. Thus, cZ E K >0. We now show that ct E K >0. The argument is similar to the previous one. Clearly, for some u E U+, we have ue = b'. By [L3] , the monomials en = erN ... ej: /«nt)! ... (nv)!) (for various n = (n}, ... , nv) E NV) span U+, hence if we write ene = E{3 d~,n{3 (where (3 runs through B>. and d~,n E K), we have d~"n i- 0 for some n. Now + t A '"'" I nl I n" h + '"'" I nl I n" d' 9 ac s on >. as wn at ... av en ence cb' = wn al ... av b',n·

Using [L2, 22.1.7], we have d~"n E K?o for all n. Since d~"n i- 0 for some n, we have db' ,n E K >0 for some n and hence En a~ nl ... a~ n" d~, ,n E

K>o. Thus, c,;, E K>o. The proposition is proved.

4. Over real numbers: G?:.o is closed in G

4.1. In §4-§8, we assume that K = Rand K>o = {a E Ria> OJ.

We will often use topological notions concerning real algebraic vari­eties; they will always refer to the standard topology coming from the euclidean structure of R.

Proposition 4.2. (a) u~o is a closed subset of U+.

(b) U;o is a dense subset of U~o.

We prove (a). Let it. i 2 , ••• iN be as in 2.10, so that U;o is the image of the continuous map q : R~o -t U+ given by -

q(al' a2, . .. , aN) = Xil (at}Xi2 (a2) ... XiN (aN).

It suffices to show that q is proper. Let r : U+ -t RI be as in 2.17. Since r is continuous, in order to show that q is proper, it suffices to show that

Total Positivity in Reductive Groups 545

the composition rq : R~o - RI is proper. In other words, it suffices to show that, given ti E R-;'o (i E f), the set

{(al,a2, ... ,aN)ER~01 L ak$ti ViEf} k:i,,=i

is compact. This is clear since this set is a product of simplices; (a) is proved.

We prove (b). Since U;;o = q(R~o), U:;o = q(R~o) (with q as above) and R~o is dense in R~~, it follo~s that U:;o is dense in U;;o. The proposition is proved. - -

Theorem 4.3. G?o is a closed subset of G.

We first show that

(a) G?o c U-TU+

where G>o is the closure of G>o in G. Assume that (a) is known when G is repl~ed by a. We have G?o C a?o n au hence

so that (a) holds when G is replaced by C. From this one can deduce by a routine argument that (a) holds for G. Thus, to prove (a) it suffices to consider the case of a instead of that of G. Let A = (Aj) E N J. Assume that A is regular, that is, that Aj > 0 for all j. Let (A.\, fJ.\) be as in 3.1. The function 9 1-+ llb Sb(g) (notation in the proof of 3.2(b» from G to R is continuous and the image of a >0 under this function is contained in the interval [1,00) (see 3.2(b»). Hence the image under this function of

the closure G?o of a?o in a is contained in [1,00). Hence 8b(g) i- 0 for

any b E B.\ and 9 E G?o. In particular,

(b) 8(g) i- 0 for any 9 E G?o

where, for any 9 E G, we denote by 8(g) E K the (fJ.\, fJ.\)-diagonal entry of the matrix of 9 : A.\ - A.\. We show that

(c) if 9 E G does not belong to (j-T(j+, then 8(g) = O.

By the Bruhat decomposition, we can write 9 = g-mg+ where g- E (j- , g+ E (j+ and m is an element of the normalizer of T such that m ¢. T. We have g+fJ.\ = fJ.\. Moreover, mfJ.\ is an extremal vector of A.\ which is not a multiple of fJ.\. Hence mfJ.\ = cb where b E B.\ - {fJ.\} and c E K*. It is clear that g-b is contained in the span Z of B.\ - {fJ.\}. Thus we have gfJ.\ = cg-b E Z and (c) follows. Combining (b) and (c), we see that (a) holds for a and hence (a) holds for G. Now any element 9 E U-TU+ can be written uniquely as a product 9 = g-gOg+ with

546 G. Lusztig

g- E U-, gO E T, g+ E U+ and the map 9 ~ (g-, gO, g+) is continuous from U-TU+ to U- x T x U+. (Here the topology of U-TU+ is the one induced from that of G.) Let (gn)n>l be a sequence of elements in G?o which is convergent to 9 in G. By-(a), we have 9 E U-TU+ and clearly, we have gn E U-TU+ for all n. We write 9 = g-gOg+ as above and similarly gn = g;g~gt. By the continuity of 9 ~ (g-,gO,g+), we see that g; converges to g- in U-, g~ converges to gO in T and gt converges to g+ in U+. Since gn E G?o, we have g; E U:;o' g~ E T>o and gt E U:;o. Since U:;o is a closed subspace of U+ (see 4.2), we see that limn gt = g+ implies g+ E U!o' Using the analogous result for U- instead of U+, we see that g- E U:;o. Since T>o is obviously closed in T (it is the identity component ofT as a topological manifold), we see that gO E T>o. Combining these results, we see that 9 = g-gOg+ E U:;oT>oU:;o = G?o. The theorem is proved. - -

Remark 4.4. G>o is a dense subset of G>o. This follows immediately from 4.2(b) and its analogue for U-. -

Lemma 4.5. uto is open in U+.

Let iI, i 2, ... , iN be a sequence in I as in 2.10. Consider the map f : R~o --+ U+ defined as in 2.7(a). This is a continuous map from an open set in RN to RN, which is injective (see 2.7(a». Hence its image is open in R N, by Brouwer's theorem of "invariance of domain" (see, for example, [D, IV, 7.4)). By definition, this image is U:;o. The proposition is proved.

Proposition 4.6. uto is a connected component of the open subset U+ n n- of U+ (notation of 2. 13).

By 2.13, we have uto = uto n (U+ n n-). Since U:;o is closed in U+, it follows that uto is closed iIi' U+ n n-. By 4.5, uto i; open in U+ n n-. Hence uto is a union of connected components of U+ n n-. Since, by definition, uto is the image of R~o under a continuous map, it follows that uto is connected. The proposition follows.

Lemma 4.7. G>o is a connected open subset of G.

Since G>o C U+TU- and U+TU- is open in G, it suffices to show that G>o is a connected open subset of U+TU-. Now U+TU- is naturally a product U+ x T x U- as a topological space and G>o may be identified with the subset uto x T>o x U;o of this product. It remains for us to use the fact that uto is connected and open in U+ (see 4.5, 4.6), that U;o is connected and open in U- (which is similar to 4.5,4.6), and that T>o is connected and open in T (which is obvious).

Total Positivity in Reductive Groups 547

Theorem 4.8. G>o is a connected component of the open subset n+nn­ofG.

By 4.7 and 2.13, G>o is a connected open subset of n+ n n-. Since G>o is closed in G, the intersection G>o n n+ n n- is closed in n+ n n-. H;nce by 2.13, G>o is closed in n+ nn--. Thus, G>o is a connected, open and closed subset of n+ n n-. The theorem is proved.

5. Over real numbers: regularity theorem

5.1. Let 9 E (;>0. Let A = (Aj) E NJ and let (AA' 1]>.) be as in 3.1. Let P be the set of lines in the R-vector space AA. Let P~o (resp. P>o) be the set of all lines in AA that are spanned by a vector whose coordinates with respect to B>. are all in R~o (resp. all in R>o). Note that P~o is a closed simplex in P and P>o is its interior. Since all entries of the matrix of 9 : A>. ~ A>. are in R>o (see 3.2(c», the obvious action of 9 on P leaves stable P~o and P>o. We shall need the following result.

Lemma 5.2. (a) There is a unique line L = Lg E P~o such that gL = L. We have necessarily Lg E P>o. Let a E R be such that 9 = al on Lg •

Then a E R>o and there exists a unique g-stable hyperplane H c A>. such that all eigenvalues of 9 : H ~ H have absolute value < a. If L' E P>o, then L' is complementary to H. -

(b) For any L' E P~o we have limn -+oo gn(L') = Lg.

(c) Let Z~o c P~o be a non-empty closed subset such that gZ~o c Z~o. Then Lg E Z~o.

(a) is just Perron's theorem [Pj applied to the matrix of 9 which has all its entries in R>o (see 3.2(c». Since a-1g : H ~ H has all its eigenvalues of absolute value < 1 (see (a», we have limn-+oo(a-1g)n(y) = 0 for any vector y E H. Now let L' be as in (b). Let x be a non-zero vector in L'. We can write uniquely x = x' + x" where x' E Lg and x" E H. By (a) we have x fi. H hence x' =I- o. We have a-ngn(x) = x' +a-ngn(x") for all n 2': 1. Since limn-+oo(a-1g)n(x") = 0, we have limn-+oo(a-1g)n(x) = x'. Using this, x' =I- 0, and the fact that (a-1g)n(x) =I- 0 for all n, we deduce that in P we have limn-+oo(a-1g)n(L') = Lg. Now a-1g = 9 as maps P ~ P hence limn -+oo gn(L') = Lg and (b) is proved. We prove (c). Let L' E Z>o. For all n 2': 1, we have gn L' E Z>o. Since the sequence gn L' converges in P to Lg (see (b» and Z~o is clo;d, it follows that Lg E Z~o. The lemma is proved.

5.3. Let 13 be the set of all Borel subgroups of (; defined over R. We regard 13 as a real algebraic variety. For any B E 13, let LB E P be the unique B-stable line in A>.. The image Z of this map is a closed, real algebraic submanifold of P. Let Z~o = Z n P~o.

548 G. Lusztig

Lemma 5.4. We have Lg E Z~o.

Clearly, Z>o is a closed subset of P>o. It is non-empty: for example, it contains the line spanned by 7]).. (which is the same as LjJ+). Since gLB = LgBg-l for any B E B, we see that 9 : P -- P leaves stable Z, hence it leaves stable Z~o. Hence 5.2(c) is applicable and the lemma follows.

5.5. We choose ,\ in the previous analysis to be regular (as in 4.3). It is known that in this case, the map B -- Z given by B -- L B is an isomorphism of algebraic varieties. By 5.4, there exists B E B such that Lg = LB' Since gLg = Lg, we have LB = gLB = L gBg-l hence B = gBg-l, hence 9 E B. Let gs E B be the semisimple part of g. We can find a R-split maximal torus S of B such that gs E S. Let x be a non-zero vector in Lg. Since g(x) = ax (see 5.2(a», we have gs(x) = ax. Let m E normalizer(S). Applying m to the previous equality, we obtain mgs(x) = am(x). Hence, if mgs = gsm, then gsm(x» = am(x). By 5.2(a), the a-eigenspace of gs : A).. -- A).. is exactly L g , hence m(x) E L g •

Since Lg = LB, we have m(LB) = LB. Since'\ is regular, the lines m'(LB ) (for various m' E normalizer(S)/S are distinct (they are different weight spaces of S). Hence from m(LB) = LB we can deduce that mE S. Thus, we have shown that m E normalizer(S), mgs = gsm imply m E S. It follows that gs is a regular semisimple element of S. This implies that 9 =gs·

The following theorem and its corollary generalize a result of Gant­macher and Krein [GK,KG) (which was concerned with the special case where G = GLn).

Theorem 5.6. Let 9 E G>o. There exists a unique R-split maximal torus of G containing g. In particular, 9 is regular and semisimple.

We have just proved this for G instead of G. We now prove that the theorem holds for G instead of G. Let 9 E G>o. Under the imbedding G c G, the set G>o becomes a subset of G>o. Since the theorem is known for G, there is a unique R-split maximal torus S of G containing g. Since G is simply connected, the centralizer of 9 in G is equal to S. Hence the centralizer of 9 in G is S n G. Similarly, the group of C-points of the centralizer of 9 in G(C) is S(C) n G(C); this consists of semisimple elements and is necessarily a maximal torus. Hence 9 is regular semisimple in G and its centralizer is the R-split maximal torus SnG. Thus, the theorem holds for G. We prove that the theorem is true for G. We can write 9 in the form 9 = Z7r(9) where 7r : G -- G is as in 1.6, 9 E G>o and Z E T>o is in the centre of G. Since the theorem holds for 9 E G, we see immediately that it also holds for 9 E G.

Total Positivity in Reductive Groups 549

Corollary 5.7. Let 9 E G'!:8. There exists a unique R-split maximal torus of G containing g. In Particular, 9 is regular and semisimple.

By definition, there exists m ~ 1 such that grn E G>o. The identity component of the centralizer of 9 (over C) is contained in the identity component of the centralizer of grn (over C) and this is an R-split max­imal torus (see 5.6). The containment above is clearly an equality; the corollary follows.

5.8. Let 9 be the Lie algebra of G (over R). Let dXi, dYi E 9 be the derivatives at 0 of Xi, Yi and let t be the Lie algebra of T. Let 9>0 be the subspace of 9 consisting of the vectors t + Ei(aidxi + bidYi) ;here t E t, ai E R~o, bi E R~o. Let 9>0 be the subspace of 9 consisting of the vectors as above such that ai E R>o, bi E R>o for all i. (These are the Jacobi elements in the terminology of Kostant [Ko].)

One should think of 9~0, 9>0 as the Lie algebra analogues of G~o, G>o (although 9~0,9>0 have much smaller dimension than G~o,G>o). (This is justified by Loewner's results for G Ln in [Lo].)

The Lie algebra analogue of Theorem 5.6 is then the following theorem of Kostant [Ko]: any element in 9>0 is regular, semisimple and R-split.

Note that

(a) for any X E 9~0, we have exp(x) E G~o.

This is asserted for GLn{R) in [Lo]. In the general case, (a) can be proved as follows. Using the known formula

(b) exp(y + z) = limn->oo«exp(yjn) exp(zjn»n (see [B, Ch.3, §6, no.4, Prop.8]) for y, z E 9, and the fact that G~o is closed in G (see 4.3) we see that if (a) holds for x = yand for x = z, then it also holds for x = y + z. It is therefore enough to prove (a) in the case where x is either in t or is one of the elements dXi, dYi, in which case (a) is obvious.

It seems likely that

(c) for any x E 9>0, we have exp(x) E G>o. This would imply that Kostant's theorem cited above is a consequence

of Theorem 5.6. The following result makes (c) plausible.

Proposition 5.9. We have

(a) exp(Ei aidxi) E U;o,

(b) exp(Ei bidYi) E U;o, (c) exp(Ei(aidxi + bidYi» E G>o,

where ai, bi (i E I) are in R>o.

We will prove this in the case where G = G. (The general case reduces to this by descent.) By 5.8(a), the elements in (a),(b),(c) are contained in Uio' U:;'o' G~o, respectively. Hence, by 2.13, it is enough to show that

550

(d) the elements in (a),(c) are contained in n-, (e) the elements in (b),(c) are contained in n+.

G. Lusztig

Since (d),(e) are symmetric, it is enough to prove (e). More generally, it is enough to verify the following statement:

(f) Let x = Ei(C;dxi + bidYi) where bi > 0 and Ci 2: 0 for all i. Then 9 = exp x E n+. Let ~ E A>. be as in the proof of 3.2, where>. is regular. We can find a unique bE B- such that br]>. =~. We can write

(g) 9TJ>. = Ep~o(l/p!)-l(Ei(c;ei + bdi))PTJ>.· We now use the following three facts: (1) any monomial in the ei,!i

(an element of U) applied to TJ>. is a linear combination with coefficients in N of elements in the canonical basis of A>.; (2) any monomial in the Ii (an element of U-) is a linear combination with coefficients in N of elements in the canonical basis of U-; (3) there exists some monomial in the Ii in which b appears with coefficient > O. .

Using these three facts and (g) (together with our assumption on bi, Ci) we see that

(h) 9TJ>. = q~ plus a linear combination of elements of B>. other than ~. (Here, q E R>o.)

We are now ready to prove (f). We certainly have 9 E B+mB- for some m in the normalizer of T. We must show that m represents woo Assume that this is not so. We write 9 = umu' where u, u' E U+. Then mu'TJ>. = mTJ>. is a multiple of an extremal weight vector other than ~ (by our assumption on m) hence applying u to it gives a linear combination of vectors in weight spaces other than that of~. Thus,9TJ>' is a linear combination of vectors in weight spaces other than that of ~. This contradicts (h). The proposition is proved. 1

5.10. Using 5.9 and 4.6, 4.8, we obtain the following alternative defini­tions of uto, U:;;o, T>o, G>o.

uto may be characterized as the connected component of U+ n n­containing the elements 5.9(a);

U:;;o may be characterized as the connected component of U- n n+ containing the elements 5.9(b);

T>o is the connected component of T containing 1;

G>o may be characterized as the connected component of n+ n n­containing the elements 5.9(c).

Using 4.2, 4.3, 4.4, we obtain the following alternative definitions of U~o,G~o:

l(Added April 30, 1994.) As A. Astashkevich pointed out to me, a proof similar to that of 5.9(c) above shows that 5.8(c) holds in general.

Total Positivity in Reductive Groups

U~o may be characterized as the closure of U;o in U+;

G~o may be characterized as the closure of G>o in G.

6. Over real numbers: unipotent elements in G?o

551

6.1. Among the (real) unipotent elements in G, very few are contained in G?o (roughly a set of half dimension). (See 6.4 below.)

Lemma 6.2. Let 9 E (;>0 be a unipotent element. Let A = (Aj) E NJ. There exists a total order on B,x such that the matrix of 9 : A,x -+ A,x (with respect to B,x) is upper triangular (for this order) with all diagonal entries equal to 1.

We recall a theorem of Frobenius [F] which generalizes Perron's the­orem [Pl. Let V be an R-vector space with a fixed finite basis and let A : V -+ V be a linear transformation whose matrix in the given basis has all entries in R?o. We say that A is reducible if we can find a parti­tion of the basis into two non-empty subsets so that the elements in the first subset span an A-stable subspace of V; otherwise, we say that A is irreducible. The theorem of Frobenius asserts that, if A is irreducible and has trace in R>o, then A has a simple eigenvalue a E R>o such that all other eigenvalues of A have absolute value < a.

If b, b' E B,x, we write b ~ b' if there is a sequence b = bo, bl , ... ,bt = b' such that bsH appears with non-zero coefficient in gbs for s = 0,1, ... , t-1. This is a preorder on B,x. This gives rise to an equivalence relation", on B,x where b '" b' means b ~ b' and b' ~ b. The set of equivalence classes inherits from ~ a partial order. We choose a total order refining this partial order. Thus we can write the equivalence classes in a sequence PI, P2 , ••• , Pt where the following holds: if b E Ps , then gb is a linear combination of basis elements in Ps U Ps+I U ....

Let V>s be the R-subspace of A,x spanned by Ps U PsH U ... ; let Vs = V?s/V?s-:;"l where V?t+l = O. Note that Vs has a natural basis (the image of Ps) and that 9 leaves stable V>s and induces a linear transformation on v;. whose matrix with respect to the natural basis has all entries in R?o (by 3.2(a». This matrix is irreducible (in the sense of the definition above) and its trace is in R>o (by 3.2(b». On the other hand, all its eigenvalues must be equal to 1 since 9 : A>.. -+ A>.. is unipotent. Applying the theorem of Frobenius to this matrix, we see that Vs is necessarily one dimensional, or equivalently, each Ps consists of a single element. The lemma follows.

Proposition 6.3. Let 9 E G>o be a unipotent element. There exists an element m in the normalizer ~fT such that 9 E mU+m- l .

We may regard 9 as a unipotent element in (;?o, fixed by (T. Let A be as in Lemma 6.2. We assume that A is regular. By Lemma 6.2, we can arrange the elements of B,x in a sequence bl , b2, . .. such that

552 G. Lusztig

(a) gbs - bs is a linear combination of bs+l, baH,' .. for s = 1,2, ....

Let D be the set of all g' E G such that (a) holds with 9 replaced by g'. Clearly, D is an algebraic subgroup of G whose image in Aut(A.>.) is unipotent. Hence the identity component Do (in the algebraic sense) of D is a unipotent group. Now 9 E Do since 9 E D and 9 is unipotent. Since T acts diagonally in the basis B.>., we see that T must normalize D hence also Do. The fixed point set Dg is then a unipotent group (which may be regarded as a subgroup of G) which contains 9 and which is normalized by T. Consider the action of T(C) by conjugation on the variety of all Borel subgroups of G(C) that contain Dg(C). (This variety is clearly projective and is non-empty since Dg(C) is unipotent.) By Borel's fixed point theorem, this action has some fixed point. Thus there exists a Borel subgroup of G(C) that contains Dg(C) (hence also g) and contains T(C). This Borel subgroup is necessarily of the form mB+(C)m-l where mEG normalizes T. The proposition follows.

6.4. Let C be a unipotent conjugacy class in G(C). Let C~o = C(R) n G~o. It is known that dim(C(R) n mU+m-1 ) :5 dimC(R)/2 for any m. Hence from 6.3, we see that C>o is a space of dimension :5 dimC(R)/2. (It may happen that C>o is empty; this is so, for example, if C is the subregular unipotent class in 808.)

6.5. For G = 8L2 the set of unipotent elements in G~o is U:;o U U;o­For G = 8L3 the set of unipotents in G~o is U:;o U U;o U A-where-A consists of all matrices of the form --

(1 0 0) a 1 b o 0 1

and with a, b E R~o.

This result is explained by the following theorem.

Theorem 6.6. The set of unipotent elements in G~o is the (disjoint) union of the sets

for various w, w' in W such that the supports of wand w' are disjoint (see 2.16).

HW HW Let "G>o be the set of unipotent elements in G>o . By 2.17(c),

the sets "G~t' (for various H c I, H' c I) form a p~rtition of the set of unipotent elements in G~o. Hence it suffices to prove the statements (a),(b) below.

( ) IfHnH' 0 th "GH,H' U+,HU-,H' U-,H'U+,H. a = , en ~o = ~o ~o = ~o ~o'

Total Positivity in Reductive Groups

(b) if H n H' # 0, then uG~t' = 0. We shall prove the following statement:

(c) if 9 E uG~oH', then H n H' = 0,

553

by induction on the semisimple rank of G. (If G is a torus, this state­ment is obvious.) Let P be the (parabolic) subgroup of G generated by T, by the xi(R) for i E H and by Yi(R) for i E I. Let 11" : P ---. L be the reductive quotient of P. Then L is naturally a group of the same kind as G, with a pinning in which the role of I is played by H. Now 1I"(g) is a unipotent element of L and from the definitions, we have 1I"(g) E L~oHnH'. If P # G, then we may apply the induction hypothesis to 1I"(g) ELand we see that H n (H n H') = 0 hence H n H' = 0, as desired. Thus, we may assume that P = G, so that H = I. Now let Q be the (parabolic) subgroup of G generated by T, by the xi(R) for i E I and by Yi(R) for i E H'. Let 11"' : Q ---. M be the reductive quotient of Q. Then M is naturally a group of the same kind as G, with a pinning in which the role of I is played by H'. Now 1I"/(g) is a unipotent element of M and from

the definitions, we have 1I"'(g) E M!!~·H'. By 2.19 applied to M, we have 1I"/(g) E Mg'l) and by 5.7, we have that 1I"/(g) is regular semisimple in M. Since it is also unipotent, it follows that M is a torus and H' = 0. Hence (c) is proved. Now (b) dearly follows from (c).

We prove (a). Let U+·H be the subgroup of U+ generated by UiEHXi(R) and let U-·H' be the subgroup of u- generated by UiEH'Yi(R). Since H n H' = 0, the unipotent groups U+·H, U-·H' are normalized by T and commute with each other so that U+·HU-·H' = U-·H' U+·H is a unipotent subgroup of G normalized by T. Hence U+·HTU-·H' is a solvable group with unipotent radical U+·HU-·H' (which is precisely the set of unipotent elements in U+·HTU-·H'). It follows that any unipo-t t I t f GH.H' U+·HT U-·H'. t· d· U+·HU-·H' d en e emen 0 >0 = >0 >0 >0 IS con ame m >0 >0 an

conversely, any el~ent in U~oHU:;'oH' is unipotent. This ~rove; (a).

7. Over real numbers: curves in U-This section contains some preparatory results needed in §8.

Proposition 7.1. Let u E U:;o and let u' E U-. There exists t E T >0 such that t-Iutu' E U:;o.

We may regard u (resp. u') as an element of U:;o (resp. U-), fixed by a. Let E be an indeterminate. Let L be the set of all f E R( E) such the power series expansion of f at 0 is of the form Ln>-I anEn where

a_I E R>o. Bya curve in U- we mean a rational map-R ---. U-; this is the same as an R(E)-rational point of U-. A curve in U- is said to

554 G. L'USztig

be admissible with respect to a reduced expression sjt sh ... Sjv for Wo in W if it is of the form Yjt (ft)Yh (h) ... Yjv (fll) where each /8 E L. Then the curve is admissible for any other reduced expression of woo (To see this, we only have to compare two reduced expressions which differ by a simple application of the braid group relations; we then use the fact that, if ft, 12, fa are in L, then so are hfal(ft + fa), ft + fa, fthl(ft + fa).)

If p is an admissible curve as above, then by specializing € to be a real number > 0, very close to 0, each /8 specializes to a real number > ° and therefore, p specializes to an element of (;:;0.

If p is an admissible curve and u' is an element of if-, then pu' is an admissible curve. Indeed, since if- is generated by elements Yj(c) with j E J, c E R, it suffices to check that PYj (c) is an admissible curve. We choose j}'i2, ... ,jll as above such that jll = j. We then have p = Yjt (ft) ... Yjv-l (fll-t)Yjv(fll) where each /8 ELand Pyj(c) = Yjl (ft) ... Yjv-l (fll-I)Yjv(fll + c); we now use the fact that /11 + eEL.

For each real number f > 0, we can find an element tf E 1'>0 such that tf is fixed by 0" and

t;-IYj(a)tf = Yj(C1a)

for all j E J, a E R. We write u in the form u = YiI (al)Yh(a2) ... Yjv(all ) with j},j2, ... ,jll fixed as above and with at. a2, ... , all in R>o. For any f > 0, we have t-;lutfu' = pu' where

p = Yjt (f-Ial)Yh(f-Ia2) ... Yjv(f-Iall ).

We may regard p as an admissible curve; then by the argument above, pu' is again an admissible curve and by specialiZing f to a real number > 0, very close to 0, pu' specializes to an element of if:;o. Thus we have t-;lutfu' E if:;o. This is an identity in G. Projecting it to G, we obtain an identity which proves the proposition.

Corollary 7.2. Let u E U:;o, U E uto. There exists t' E T >0 such that uut'u- l E G>o.

We have uu = Uitlul with UI E U:;o, tl E T>o, UI E uto. (See 2.12.) By 7.1, we can find t E T>o such that t-IUltU-1 E U:;o. We have

uutu- l = UlltUltU-1 = Ulttt(CIUltU-I ) E UtoT>oU:;o = G>o.

The corollary is proved.

Remark 7.3. t' E T>o in the previous corollary may be chosen such that X~(t') > 1 for all i . Indeed, in the previous proof we may take t = 1I"(tf ) (11" as in 1.6, tf as in 7.1) with f a real number> 0, very close to 0, and then t' = lt1l"(tf ). We have

X~(t') = X~(lt)X~(1I"(tf» = X~(lt)f-l.

Since X~(tt) > 0, by chooosing f > ° sufficiently close to 0, we will have XWI)f > 1, hence X~(t') > 1 for all i, as asserted.

Total Positivity in Reductive Groups 555

8. Over real numbers: the subsets B>o, B?o of the flag manifold

8.1. Let B the (real) flag manifold of G, that is, the real algebraic variety whose points are the Borel subgroups of G defined over R. We will show that B has a remarkable closed subset B>o. In the case where G = 8L2 , this is a closed interval with ends B+~ B-. In the general case, our definition of B?o will be initially for G instead of G and will be asymmetric in iJ+, iJ-; these deficiencies will be removed in 8.8.

By definition, 8?0 is the closure of the set {uiJ+u-1Iu E ir;-o} in 8 (8 as in 5.3).

Lemma 8.2. Let gE G>o. There exists a unique Borel subgroup B in 8?0 such that 9 E B.

Let ,\ E NJ. In the setup of 5.3, we replace Z?o by the image Z;o of

8'20 under the map p : 8 -+ P given by B 1-+ LB. Since p is proper,-Z~o coincides with the closure of {u1J>.lu E ir;-o} in P. Therefore, Z~o C P?o. (See 3.2(a).) We show that Z~o is g-stable for 9 E G>o. It suffices to

show that {u1J>.lu E ir;-o} is a g-stable subset of P. This follows from the inclusions G>oir;-o C G>o c ir;-oB+ (see 2.12). We may apply 5.2(c) to Z~o instead of Z?o and we deduce that Lg E Z~o. We now assume that

,\ is regular. From Lg E Z~o we deduce that Lg = LB for some BE 8?0. Since the stabilizer of LB in G is exactly B and the stabilizer of Lg in G contains g, we have 9 E B. Now let B' be any Borel subgroup in 8?o such that 9 E B'. Then LBI E Z;o hence LBI E P?o. Since 9 E B', the line LB' is g-stable. By the definition of Lg, we then have LB' = Lg. Hence we have LBI = LB and B = B'. The lemma is proved.

Remark 8.3. If g, B are as Lemma 8.2, then the lines Lg, LB in A>. co­incide for any ,\ E NJ. Indeed, as above, LB is a g-stable line in P?o hence it coincides with Lg by the definition of Lg.

Lemma 8.4. Let 9 E G>o and let B be as in 8.2. There exists u E ir;-o such that B = uiJ+u-1. -

We shall deduce the lemma from the following statement.

(a) There exists u E ir- such that B = uiJ+u-1. Using (a) and the definition of B, we see that uiJ+u-1 is contained in the closure of {u' iJ+u,-llu' E ir;-o} in 8. Since the map ir- -+ 8 given by u 1-+ uiJ+u-1 is a homeomorphism of ir- onto its image, it follows that u is contained in the closure of ir;-o in ir- and this equals ir:;o' by 4.2. -

It remains to prove (a). Assume that (a) does not hold. Then, by Bruhat decomposition, there exist u E ir- and m E normalizer(T) - T

556 G. Lusztig

such that B = umiJ+m-1u-1• We choose A E NJ regular. Since the lines LB, Lg in A>.. coincide and umTJ>.. E LB, we have umTJ>.. E Lg• By our assumption on m, the vector mTJ>.. is in a weight space other than the highest one; applying to it u E (;- will take this to a sum of even lower weight spaces; in particular, the coefficient of TJ>.. in umTJ>.. (as a linear combination of elements in B>..) is zero. On the other hand, a non-zero vector in Lg must have all its coefficients different from zero (in fact all > 0 or all < 0) by 5.2(a). This is a contradiction. The lemma is proved.

Lemma 8.5. Let 9 E 0>0 and let B be as in B.2. There exists u E (;:;0 such that B = uiJ+u-1.

Let u E (;~o be as in Lemma 8.4. By 2.10, we can write

where jl,h, ... ,jv is a sequence in J such that sit sh .. , Sj" is a reduced expression for Wo in Wand aI, a2, ... ,av are in R~o. It suffices to show that all as are in R>o. Assume that as = 0 for some s.

One can now verify that the monomials fj/ fj;2 ... fJ'.," (for various nl, n2, ... ,nv E N such that ns = 0) span a proper subspace of U-. It follows that there exists an element b of the canonical basis of U- that is not contained in this subspace.

We can choose A E NJ with large enough coordinates so that bTJ>.. =f 0 and so that the subspace of A>.. spanned by the elements 1';/ fj;2 ... fj;TJ>.. (for various nl, n2, ... ,nv E N with ns = 0) does not contain the element bTJ>.. E B>... This subset is spanned by a subset of B>.. (we use [L3, 4.2]). It follows that bTJ>.. appears with coefficient zero in UTJ>.. (expressed as a linear combination of elements in B>..). Since the lines LB, Lg in A>.. coincide and UTJ>.. E LB, we have UTJ>.. E Lg. But as in the proof of Lemma 8.4, a non-zero vector in Lg must have all its coefficients different from zero. This is a contradiction. The lemma is proved.

8.6. The results above have analogues in which iJ+, iJ- change roles. The proofs are entirely analogous to the ones above. We state these results.

(a) Let 9 E 0>0. There exists a unique Borel subgroup B' in the closure of the set {u' iJ-u,-llu' E (;;t"o} in 13 such that 9 E B'.

(b) If g, B' are as in (a), then the lines Lg, LB' in A>.. coincide for any AE N J •

(c) If g, B' are as in (a), then there exists u' E (;;t"o such that B' = u'iJ-u,-l.

From Corollary 7.2, we deduce the following result.

(d) If u E (;:;0' then uiJ+u-1 contains some element of 0>0.

Total Positivity in Reductive Groups 557

Again, by interchanging the roles of jJ+, jJ-, we deduce the following result.

(e) If u' E lito, then u' jJ-u,-l contains some element of 0>0.

Theorem 8.7. The subsets {uB+u-1Iu E U;o} and {u' B-U,-llu' E

U;o} of 8 coincide.

Let u E U;o. We regard u as a a-fixed element of li;o. By 8.6(d), there exists 9 E 0>0 such that 9 E ujJ+u-1 • Let B' be associated to 9 as in 8.6(a) and let B be associated to 9 as in 8.2. From 8.3 and 8.6(b) we see that for any ,\ E NJ, the lines LB, LB' in A)" coincide; hence B = B'. By the definition of B, we have B = ujJ+u-1 • By 8.6(c), there exists u' E li;o such that B' = U' jJ-U'-l. Since B = B', we have ujJ+u-1 = u' jJ-u'-l. Since ujJ+u-1 is a-stable, the same holds for u' jJ-u'-l. Thus, U' jJ-U,-l = 0'( u')jJ-0'( U')-l. From this we deduce, using Bruhat decomposition, that u' = o'(u'). Hence u' may be regarded as an element of U;o. Taking fixed point sets of 0' in the two sides of the equality ujJ+u-1 = u' jJ-U'-l, we deduce uiJ+u-1 = u' iJ-u'-l; this clearly implies uB+u-1 = u' B-U'-l. Thus, the first set in the theorem is contained in the second set. The reverse inclusion is entirely analogous. The theorem follows.

8.8. The two sets whose equality is asserted in the theorem will be denoted by 8>0. By definition, 8>0 is the closure of 8>0 in 8. Note that the definitions of 8>0,8>0 are symmetric in B+, B-. Replacing G by o we get analogous subs;ts 8>0,8?0 of 8. (This last definition of 8?0 clearly agrees with the one given in 8.1.)

The flag manifolds 8,8 are related as follows: we may identify 8 with the set of fixed points 8 tT of 0' on 8. (To B E 8 corresponds B' E 8 tT where 7r-1 (B) = B'tT, 7r as in 1.6.) Under this identification, 8>0 clearly corresponds to (8)0)tT; taking closure, we deduce that 8>0 corresponds to (8?0)tT.

Theorem 8.9.

(a) Let 9 E G>o. There exists a unique Borel subgroup B E 8?0 such that 9 E B. We have automatically BE 8>0.

(b) Let 9 E 0>0 and let B be the unique Borel subgroup in 8>0 such that 9 E B. For any,\ E N J , the lines Lg,LB in A). coincide. -

(c) The map G>o -+ 8>0 given by 9 ~ B (as in (a)) is continuous.

(a) has been already proved for 0 instead of G; (b) has also been proved already. Now (a) for G follows immediately from (a) for 0 by the identifications in 8.8. Moreover, (a) for G follows immediately from (a) for G. We prove (c). We may assume that G = O. We choose ,\ E N J

558 G. Lusztig

regular. It is enough to show that the map 9 I-t L9 is a continuous map G>o --t P>o (notation of 5.1,5.2). This follows from the fact that the Per­ron eigenspace of a. matrix with all entries in R>o depends continuously of that matrix. The theorem is proved.

Corollary 8.10. Let 9 E G>o. There exist u E U:;o, u' E U:to and t E T>o such that 9 = uu'tu-1 • Moreover, u, u', t are uniquely determined and we have xW) > 1 for all i.

By 8.9{a), we have 9 E uB+u-1 for some u E U:;o. We can write 9 = uu'tu-1 with U' E U+ and t E T. We have

(see 2.12). Thus uu't = Ulu~tl

where Ul E U:;o, tt E T>o, u~ E U:to. It follows that t = tt, u' = u~ hence t E T>o, u' E U:to. This proves existence. We show uniqueness. Assume that we have also 9 = uu'tu-1 with u E U:;o, u' E U:to and t E T>o. Then uB+u-1, uB+u-1 are two Borel subgroups in B>o containing g. By 8.9(a), we have uB+u-1 = uB+u-1 hence u = u. From this, the equalities u' = u', t = t follow immediately.

It remains to show that xHt) > 1 for all i. In any case, since 9 is regular semisimple (see 5.6) and conjugate to u't, the element u't must be regular semisimple, hence t is regular semisimple. Hence we have xW) # 1 for all i. The map G>o --t {t' E T>olxHt') # 1 Vi} given by 9 I-t t (as above) is continuous. (This follows from the fact that the map 9 I-t U, U as above, is continuous and this, in turn, follows from 8.9(c).) Hence, if i E I, the map Pi : G>o --t {a E R>ola # ' I} given by 9 I-t x~ ( t) (as above) is continuous. The image of Pi is contained in either {a E R>ola > I} or in {a E R>ola < I}, since G>o is connected. To see that the image of Pi is contained in the first of these two sets it is enough to see that Pi(9'), for one particular g' E (;>0, is contained in that set. Let U2 E U:;o, u~ E U:to. By Remark 7.3, we can find t2 E T>o such that 92 = U2u~t2u21 belongs to G>o and such that X~(t2) > 1. We have P(92) = t2 and the proof is complete.

Corollary 8.11. Let 9 E G~o. There exists B E B~o such that 9 E B.

The first projection G>o x B>o --t G>o is continuous and proper, since B>o is compact. Hence it; restrktion {(9, B) E G>o x B>olg E B} --t G>o is -proper (it is restriction to a closed subset). Theref~re its image is-a closed subset of G~o. By 8.2, this image contains G>o, which by 4.4 is a dense subset of G~o. Hence this image is the whole of G~o. The corollary follows.

Total Positivity in Reductive Groups

Proposition 8.12. Let 9 E G?o.

(a) If BE 8>0, then gBg-1 E 8>0.

(b) If BE 8>0, then gBg-l E 8>0. - -

559

To prove (a) we may assume that 9 is a generator of G?o. Thus, we may assume that 9 E U;;o or 9 E uto or 9 E T>o. Assume first that 9 E U;;o. We write B = uB+u-1 where ~ E U;;o. We have gu = Ul E U;;o (see 2~12) hence gBg-1 = uIB+Ul1 E 8>0. The case where 9 E uto is entirely similar (we write B = U' B-u,-l where u' E uto). The -case where 9 E T>o is immediate. Thus (a) is proved. Now (b) follows from ( a) by taking closure.

Proposition 8.13. Let w E Wand let m be a representative of w in the normalizer ofT. Then mB-m-1 belongs to 8?0.

We prove this by induction on lew). Since 1 is in the closure of uto (in U+), we see that B- is in the closure of {uB-u-1Iu E uto} = 8>0. Thus, B- E 8>0. Thus, we may assume that w =f 1 and that the re­sult is known for w' where w' = SiW satisfies lew') < lew). Let m' be a representative of w' in the normalizer of T. Using m' B-m,-l E 8>0 (in­duction hypothesis) and 8.12(b), we see that xi(a)m' B-m'-lxi(~)-l E 8?0 for any a E R>o. We have Xi (a)m' B- = Yi(1/a)sim' B- hence Yi(1/a)sim' B-m'-ls;lYi(1/a)-1 E 8?0 for any a E R>o. Taking the limit as a --+ 00, we obtain Sim' B-m,-ls;l E 8?0, hence mB-m-1 E 8?0. This completes the proof.

Proposition 8.14. Let 8+ (resp. 8-) be the open subset of 8 consisting of the Borel subgroups that are opposed to B- (resp. to B+). The space 8>0 is one of the connected components of the open set 8+ n 8- .

From the definition it is clear that 8>0 is an open connected non-empty subset of 8+ n 8-. It remains to show that 8>0 is closed in 8+ n 8- . Since 8>0 is closed in 8, it suffices to show that 8>0 n 8+ n 8- = 8>0. Let B E 8?0 n 8+ n 8-. Since B E 8-, we ha~e B = uB+u-1 for a unique u E U-. Since B E 8>0, we see as in the proof of 8.4 that u E U~o. Since B E 8+, we see that u E n+. Since u E U~o n n+, we have u E U;;o, by 2.13(a). Thus, BE 8>0. The proposition is proved.

Remark 8.15 For any w ::; w' in W (for the standard partial order), we denote by 8>0( w ::; w') the subspace of 8>0 which consists of all B E 8>0 such that (ii-, B) are in relative positio~ w' and (B, B+) are in relati~e position w-1wo. The subspaces 8>0( w.::; w') form a partition of 8>0. We will show elsewhere that this is-a cell decomposition; more precisdy, 8?0( w ::; w') is diffeomorphic to R~-;;")-I(W). The largest stratum (cor­responding to w = 1, w' = wo) is 8>0 and the strata corresponding to

560 G. Lusztig

w = w' are the points described in 8.13. For example, for G = SL2 there are three strata and for G = SL3 there are 19 strata.

8.16. Corollary 8.11 shows that for any 9 E G~o, the space {B E B~olg E B} is non-empty. It is obviously compact, and we conjecture that it is contractible.

Proposition 8.17. Assume that G = G and let us fix a regular A E N J .

(a) If BE B, then we have B E B>o if and only if the line LB in A>. belongs to P>o (notation of 5.2).

(b) If B E B, then we have B E B~o if and only if the line LB in A>. belongs to P~o (notation of 5.2).

If BE B>o, then B contains some 9 E G>o (see 8.6(d)). By 8.9(b), we have LB = L9 and by 5.2 we have L9 E P>o. Thus, LB E P>o.

We prove the converse. Let y be the set of all B E B such that LB E P>o. Since the lowest weight vector and the heighest weight vector in the canonical basis of A>. appear with non-zero coefficient in a non-zero vector of LB (for BEY), it follows that B is opposed to B+ and to B-. Thus, Y is an (open) subset of B+ n B- .

Let BEY. From 3.2(c) we see that, for any 9 E G>o, we have gBg-1 E y.

Hence if we set, for r E R~o, gr = exp(r Li(dxi + dYi)) E G, then grBg;l E y. (For r > 0, we have gr E G>o by 5.9(c) and for r = 0 we have grBg;l = B.)

Applying 5.2(b) to 9 = gl : A>. - A>. and noting that gj = gn for all integers n ~ 1, we see that the sequence gnLB (n = 1,2, ... ) converges in P>o to the Perron line L91 E P>o (notation of 5.2(a)). By 8.9(a), there is a unique B1 E B>o such that L91 = LB1. Since gnLB converges to L B1 , it follows that gnBg;;l converges to B1 (in B). Since B1 E B>o and B>o is open in B, it follows that gnoBg;;ol E B>o for some integer no ~ l.

Since the map r ...... grBg;l from R~o to Y is continuous, its image is contained in a single connected component of B+ n B-. In particular, the images of 0 and that of no (namely, Band gnoBg;;ol) belong to the same connected component of B+ n B-. Since gnoBg;:;ol E B>o and B>o is a connected component of B+ nB- (see 8.14), it follows that B E B>o. This proves (a).

We prove (b). If B E B~o, then B is in the closure of B>o hence (using (a)), LB is in the closure of P>o, which is P~o. Conversely, assume that B E B satisfies LB E P~o. From 3.2(c), we see that, if 9 E G>o, then 9 maps any line in P>o to a line in P>o. In particular, we have grLB E P>o or equivalently, L9:B9;:1 E P>o, for any r > 0 (gr as above). Using (a), we deduce that grBg;l E B>o, for any r > o. Taking now the limit

Total Positivity in Reductive Groups 561

as r -+ 0 we see that B is in the closure of B>o, which is B?o. The proposition is proved.

9. Real rational functions and the zones of U~o and of G?o

9.1. In this section K and K>o are as in 2.1(d). Let J be the set of all finite sequences (it. h, ... ) in J such that sit sh ... is a reduced expression in W. The sequences (it.j2, ... ) E J such that sitsh ... is a fixed element w of W form a subset J (w) of J. We have a partition J = UwJ(w).

Following Iwahori and Tits, we shall regard J as the set of vertices of a graph in which j = (iI, h, ... ) and j' = (j~, j~, . .. ) are joined if j' is obtained from j by

(a) replacing three consecutive entries j,J,j in j (such that j f:.] are joined in the Coxeter graph) by ],j,], or by

(b) replacing two consecutive entries j,] in j (such that j f:. ] are not joined in the Coxeter graph) by],j.

By a theorem of Iwahori and Tits, the connected components of the graph J are precisely the subsets J (w) for various w E W.

Given j,j' that are joined in J (both of length, say, n) we define a map

Rf: zn ~ zn

as follows. This map takes c = (Cll ... , en) E zn to c' = (c;., ... , c~) E zn which has the same coordinates as c except in the three (resp. two) consecutive positions at which j,j' differ; if (m, n,p) (resp. (m, n» are the coordinates of c at those three (resp. two) positions, the coordinates of c' at those positions are

(n + p - min(m,p), min(m,p), m + n - min(m,p» (resp. (n, m».

It is easy to check that Rf is a bijection; its inverse is Rj,. Let j be the set of all pairs (j, c) where j = (il,h, ... ) E J and

(Cl' C2, •.• ) is a sequence of integers with the same number of terms as j. We shall regard j as the set of vertices of a graph in which (j, c) and (j', c') are joined if j,j' are joined in J and Rl' (c) = c'.

9.2. We have a partition j = Uwj(w) where jew) consists of all pairs (j, c) with j E J (w). Let j f rv be the set of connected components of the graph j (a quotient set of j) and let j (w) f rv be the set of connected components of the full subgraph jew) of j (a quotient set of j(w». We have jf "'= Uw(j(w)f rv).

562 G. Lusztig

9.3. For any (j,e) = «jl,h, ... ,jn},(cI,C2, ... ,en) E J, let li+(j,e} be the set of all elements of li+ of the form Xii (pJ)xi2(P2} ... Xjn (Pn) where P}'P2,··. ,Pn are in K>o and IPII = c}, Ip21 = C2,.·., IPnl = Cn (IPsl as in 2.1). From 2.7, we see that for a fixed j = (jl,h, ... ,jn) E J(w), we have a partition

(a) li+(w) = ucli+(j,c) where e runs over zn.

Proposition 9.4. Let j -:F j' be two joined elements of J (w) (of length n) and let e, e' be elements ofzn such that Rj' (e) = e'. We have li+(j, e} = li+ (j', e').

Using the identity 2.5(b}, we see that it is enough to verify the following statement.

Let a, b, c, a', b', c' E K>o be such that a' = bc/(a + c}, b' = a + c, c' = ab/(a + c). Then

la'i = Ibl+lcl-min(lal, Icl}, WI = min(lal, Icl}, Ic'l = lal+lbl-min(lal, Icl}.

This is immediate. The proposition is proved.

9.5. From Proposition 9.4 we see, using the connectedness of the graph J(w), that the partition 9.3(a} of li+(w} is independent of the choice of j in J(w}. Thus, this partition is intrinsic; the subsets of li+(w) which make up this partition are called w-zones. A subset of lito is said to be

a zone if it is a w-zone for some w E W, which is necessarily unique since li:;o = uwli+(w}. The zones form a partition of li:;o which refines the p;rtition li:;o = uwli+(w). -

Given j E J(w) (of length n), we consider the maps

fJ - j' N n --+ J(w)/ ""- ( set of w - zones)

where /j(e} = (j,e) and f'(j',e') = li+(j',e'). From the connectedness of the graph J(w} we see that /j is surjective and from 9.3(a) we see that the composition f' fj is bijective. It follows that both f' and fj are bijective. Thus, (a) and (b) below hold.

(a) The assignment (j',e') f-t li+(j',e') defines a bijection between the set J / "" and the set of zones of lito; this restricts to abijection between

the set J (w) / "" and the set of w-zones.

(b) For any j E J(w) (of length n), the assignment e f-t (j, c) defines a bijection between Nn and J(w)/ "". 9.6. Nowa: li:;o ---t li:;opermutes among themselves the zones of li:;o. Consider the zones that-are stable under a. Taking the fixed point sets of

Total Positivity in Reductive Groups 563

a on each such zone, we obtain a collection of non-empty subsets which form a partition of (uto)/7 = uto. These subsets are by definition the zones of uto. These z;nes are n;-turally parametrized by the fixed point set (J / rv);; of the action of a on j / rv (induced by the obvious action of a on j). The partition of uto refines the partition uto = UWEWU+(W). The zones that are contained in U+ (w) are then naturally parametrized by the fixed point set (j (w) / rv)/7, where w E W is regarded as an element of W, fixed bya.

9.1. Entirely analogous r~sults hold when uto' xj(a), uto are replaced by u?o, Yj(a), U?'o. We obtain in the same way ;ubsets u-(j, c) for (j, c) E j, a partition of U?o into zones indexed by J / rv and a partition of U?o into zones indexed by (J/ rv)/7.

9.8. For y E Y (see 1.4), let

T(y) = {t E Tlx(t) E K>o, Ix(t)1 = (y,x) \Ix E X} C T>o.

Note that T>o = UyEyT(y). The zones of G?o are by definition the subsets Z+T(y)Z- where Z+ (resp. Z-) runs over the zones of U~o (resp. of U:;o) and y runs over Y. Using 2.3(b) we see that the zones form a partition of G?o. It is easy to check, using 1.3(a),(b),(c), that the zones of G?o are precisely the subsets Z-T(y)Z+ where Z+,Z-,y are as above.

For example, if 1= {i} and G = 8L2 , we set

Z: = {xi(a)la E K>o, lal = r}, Zt = {Yi(a)la E K>o, lal = t},

Ts = {hi(a)la E K>ollal = s} (for r,s,t E Z)

r' = r - min(r + t, 2s), s' = s - min(r + t, 2s), t' = t - min(r + t, 2s)

and

r = r' - min(r' + t', 2s'), s = s' - min(r' + t', 2s'), t = t' - min(r' +t', 2s').

Proposition 9.9. (aJ Let Z, Z' be zones of uto. Then the subset ZZ' of uto is again a zone. -

(b) The rule Z, Z' 1---+ Z Z' defines a monoid structure on the set of zones of U~o' The unit element is the zone U+(I) = {I}.

Forj E J, c E Z, we consider the zone U+(j, c) = {xj(a)la E K>o, lal = c}.

564 G. LU8Ztig

If (j,e) E J(w), and i(sjw) = i(w) + 1 then from the definitions we have U+(j, c)U+(j, e) = U+(j',e') where j' is j followed by the sequence j' and e' is c followed by the sequence e. Using this repeatedly, we see that, if Z = U+(j, e) with j = (jbh, ... ,jn) and e = (ct, C2, ••• , cn), then Z = U+(jl, cl)U+(h, C2) •• • U+(jn, en). Hence, to prove (a), it is enough to consider the case where Z = U+(j, c) for some j E J, c E Z. We have Z' C U+(w) for some w. If i(sjw) = i(w) + 1 then ZZ' is a zone by the argument above. If i(sjw) = i(w) -1 then we have Z' = U+(j,e) where j starts with j and e is as above and then clearly Z Z' = U+ (j, e') where e' = (min(c,cl),c2,c3, ... ,cn). The proposition is proved.

Proposition 9.10. (a) Let Z, Z' be zones of G>o. Then the subset ZZ' of G?o is again a zone. -

(b) The rule Z, Z' f-+ ZZ' defines a monoid structure on the set of zones of G?o. The unit element is the zone To.

This follows immediately from the definitions using the previous propo­sition.

9.11. Let M+ be the monoid (with 1) defined by the generators E.'] (j E

J, n E Z) and the relations

(a) E.,]E.j' = E.tn(n,n/) for any j E J and any n, n' E Zj

(b) E.j E.f E.f = E.'{ E.j' E.J' for any j of 3 in J that are joined in the Coxeter graph and any integers m, n, p, m', n', p' such that

m' = n + p - min(m,p), n' = min(m,p),p' = m + n - min(m,p)

or, equivalently,

m = n' + p' - min(m',p'),n = min(m',p'),p = m' + n' - min(m',p') j

(c) E.jE"f = E"fE.j for any j of] in J that are not joined in the Coxeter graph and any integers m, n.

(Note that E.J is not assumed to be 1.) It is clear that we have a unique homomorphism Pl of monoids from

M+ to the monoid in 9.9 given by E.'] f-+ U+(j, n) for all j E J, n E N. On the other hand, we have a well defined map P2 : J / "'---t M+ such that «jl ,h, ... ), (ct, C2, ••• )) f-+ E.j: E.j: . . .. By an argument similar to that in the proof of 9.9, we see that P2 is surjective. Since the composition PlP2 is a bijection (see 9.5(a)), it follows that both Pl and P2 are bijective.

Thus, the zones of U+ are naturally indexed by M+.

9.12. Similarly, the zones of U+ form a monoid under multiplication, which is isomorphic to the fixed point set (M+)<7. The zones of G?o also form a monoid under multiplication.

Total Positivity in Reductive Groups 565

9.13. A part of the regularity theorem 5.6 remains valid for our K (real rational functions). Namely, any 9 E G>o is regular, semisimple (but, this time, not necessarily K-split). Indeed, if we specialize € to a real number > 0, very close to 0, the element 9 will specialize to an element of the real points of G, which is in G>o (over real numbers), hence is regular, semisimple by 5.6. It follows that 9 itself is regular semisimple.

10. A relationship between zones and canonical bases

10.1. In this section we preserve the setup of §9. The combinatorics we used to define j / '" is extremely similar to

the combinatorics used in [L1],[L2,§42] to parametrize canonical bases of enveloping algebras. We will exploit this similarity as follows.

Let G* be the Langlands dual of G. The algebras associated to G* in the same way as U, U- are attached to G, are denoted by *U, *U-; to G* corresponds also the modified version *U of *U, as in [L2,§23]. Let *B- be the canonical basis of *U- and let *:8 be the canonical basis of *U, as in [L2,§25], for v = 1.

Let jO be the set of all (j, c) E j (wo) such that all terms of c are in N. Then jO is a union of connected components of the graph j; let jO / '" be the set of its connected components (a subset of j/ "').

By [L2, 42.1.14], jO / '" parametrizes naturally the canonical basis B­of U- (see 3.1) and by [L2, 14.4.9], the fixed point set (B-V of (J' on B­is in natural bijection with *B-.

Thus, * B - is naturally indexed by (jO / "'V hence is naturally imbed­ded in the set (j / ",)(7 which indexes the zones of U+.

In particular, the canonical basis of a simple finite dimensional alge­braic representation of G* is naturally indexed by a subset of the set of zones of U+. This should be related to the phenomenon found in [L4], according to which many features of the simple finite dimensional rep­resentations of G* (dimensions of weight spaces, multiplicities in tensor products) can be seen in terms of the perverse sheaves on the affine flag manifold of G.

Similarly, the canonical basis *:8 is naturally indexed by a subset of the set of zones of G.

11. The elements uw,p E U+

11.1. Let w E Wand let P = (Pi) E Kf. We want to associate to wand P an element uw,p E U+.

Choose a reduced expression Sil Si2 ••• Sim for w in W. For k E [1, m], we write

(a) Sil Si2 ••• Sik_l (Xi,,) = LiEf Ti,kXi

where Ti,k E N (and Ti,k =F ° for some i)j we set

566 G. Lusztig

(b) hk = EiEI ri,kPi" E K.

Proposition 11.2. The element

(aJ UW,fJ = XiI (h1)Xi2(h2) ... xi ... (hm ) E U+

is well defined, that is, it is independent of the choice of reduced expression forw.

One can check that the element in the right hand side of (a) is equal to an analogous element in U+ (w) (where we now regard w as an element of W). This reduces the proof for G to the proof of the analogous statement for G. Therefore, in the remainder of the proof we assume that G is simply laced.

Let Si~ Si2 ••• Si~ be another reduced expression for w in Wand let r~ k' hI., be defined in terms of this reduced expression in the same way as ri:k, hk where defined in terms of SiI Si2 ... Sim • We must show that

To prove this, we may assume, by the theorem of I wahori and Tits, that Si~ Si2 ••• Si~ is obtained from Si1 Si2 ... Sim by an elementary step given by a braid group relation. Assume first that ii, i~, ... ,i~ is obtained from !l, i3' ... ,im by replacing the entries i, i, i (in positions t, t + 1, t + 2)by i, i, i.

From the definitions we have

ri,k = rLk for k fj. {t, t + 1, t + 2},

ri,t = Pi, ri,t+l = Pi + P~, ri,t+2 = P~, , " , ,

ri,t = Pi' ri,t+l = Pi + Pi' ri,t+2 = Pi

for some Pi, P~ EN. Hence

hk = hI., for k fj. {t, t + 2},

ht+1 = h~+1 = ht + ht+2 = h~ + h~+2' ht = h~+2' ht+2 = h~.

Therefore in this case we are reduced to verifying the identity

which follows from the previous equalities and from 2.5(b). A similar (and simpler) argument applies in the case where ii, i~, ... , i~

is obtained from ilt i2, .. . ,im by replacing the entries i, i (in positions t, t + l)by i, i. The proposition follows.

Total Positivity in Reductive Groups 567

Proposition 11.3. If wE Wand P = (Pi) E K~o, then uw,p E U+(w} (see 2.7).

Indeed, in this case we have hk E K>o for all k E [I,m] (notation of ILl).

11.4. Assume now that w = woo Let (Pi) E KI be as in 11.1. For any i E I we define

Ti = ~ 8i1 8 i2 ••• 8i,._1 (Xi,.) k:l<k<N

i"k=i

where 8i18i2 .. . 8iN is a reduced expression for woo (Note that Ti is inde­pendent of the choice of reduced expression.) We have Ti = :Ei'EI Bi,i'Xi'

where Bi,i' E N. Set ai = :Ei'EI 8i,i'Pi' It seems likely that

(a) exp(:Ei aidxi} = UWQ,p holds.2 This, together with 11.3, would give another proof of 5.9(a}.

REFERENCES

[A] T. Ando, Totally positive matrices, Linear Alg. Appl. 90 (1987), 165-219. [B] N. Bourbaki, Groupes et algebres de Lie, Ch. 2,9, Hermann, Paris, 1972. [D] A. Dold, Lectures on algebraic topology, Springer Verlag, 1972. [FP] M. Fekete and G. P6lya, Uber ein Problem von Laguerre, Rendiconti del Circ.

Mat. Palermo 34 (1912), 89-120. [F] G. Frobenius, Uber Matrizen aus nicht negativen Elementen, Sitzungsberichte

Preuss. Akad. Wiss. (1912), 456-477. [GK] F. Gantmacher and M. Krein, Sur les matrices oscillatoires, Comptes Rendues

Acad. Sci. Paris 201 (1935), 577-579. [KG] ___ , Sur les matrices oscillatoires et completement non negatives, Com­

positio Math. 4 (1937), 445-476. [Ko] B. Kostant, The solution to a generalized Toda lattice and representation the­

ory, Adv. in Math. 34 (1979), 195-338. [Lo) C. Loewner, On totally positive matrices, Math. Z. 63 (1955), 338-340. [L1] G. Lusztig, Canonical bases arising from quantized enveloping algebras, J.

Amer. Math. Soc. 3 (1990), 447-498. [L2] ___ , Introduction to quantum groups, Birkhauser, Boston, 1993. [L3] ___ , Problems on quantum groups, Amer. Math. Soc., Proc. Symp. Pure

Math. (1994). [L4] ___ , Singularities, character formulas and a q- analog of weight multiplic­

ities, Asterisque 101·102 (1983), 208-229. [P) O. Perron, Zur theorie der matrizen, Math. Annalen 64 (1907),248-263. [S] I. Schoenberg, Uber variationsvermindernde lineare Transformationen, Math.

Z. 32 (1930), 321-322.

2(Added April 30, 1994.) Formula (a) is verified in the paper "Factorization of certain exponentials in Lie groups", by C. K. Fan and G. Lusztig, written after the present paper has been submitted.

568 G. Lusztig

(WI A. M. Whitney, A reduction theorem/or totally positive matrices, J. d'Analyse Math. 2 (1952), 88-92.

Department of Mathematics, M. I. T., Cambridge, MA 02139 and Institut Mittag-Leffler, Auravagen 17, S-18262, Djursholm, Sweden

Received October 3, 1993

The Spectrum of Certain Invariant Differential Operators Associated to a Hermitian

Symmetric Space*

Siddhartha Sahi

This paper is dedicated to Prof. Bertram Kostant with affection and admiration on the occasion

of his sixty-fifth birthday

Let G / K be an irreducible Hermitian symmetric space of rank n and let 9 = t + p+ + p_ be the usual decomposition of 9 = Lie(G)c. Let us write P, V, and W = P ® V respectively, for the algebra of holomorphic polynomials, the algebra of constant coefficient holomorphic differential operators, and the "Weyl algebra" of polynomial coefficient holomorphic differential operators on 1'-, and regard all three as K -modules in the usual way.

Let I = WK be the algebra of K-invariant differential operators on p_., and let A be the set {oX = (Ab" . ,An) E zn I Ai 2: ... 2: An 2: a}.

Then, as we show in the next section, A naturally parametrizes both the irreducible K-submodules P).. of P, as well as a certain distinguished (vector space) basis {D.d of I. The problem we consider is to determine, for A, J.l E A, the scalar eigenvalue cl'(A) by which DI' acts on P>..

Our main result, proved in Section 1, is the following characterization of these eigenvalues. For A = (Ai,' .. ,An), let us write IAI for Ai + ... + An; and set Am = {A E A IIAI:::; m}. Also, put P = (Pl,···,Pn) where Pi = d( n - 2i + 1)/2 and d is as in the next section.

Theorem 1. There is a polynomial PI' in n variables such that cl'(A) = PI'(A + p), for all A E A. Moreover, up to a scalar multiple, PI' is charac­terized by the following properties:

(a) PI' is symmetric and has degree at most 1J.l1. (b) PI' (J.l + p) is nonzero, while PI' (A + p) is zero for all other A E AII'I'

(This result was conjectured by B. Kostant during a conversation with the author).

* This research was partially supported by an NSF grant at Rutgers University.

570 S. Sahi

To indicate the power of this result we give two applications. For each k = O,l,···,n, let us put v k = (1,.··,1,0,···,0) E A (with k l's), and write v = vn .

The first of our applications is a simple proof of the Capelli identity of [KS] which is the formula for Pmv for m E Z+.

Corollary 1. Pmv(x) equalsn~ln7=:~/(Xi-Pn-j), wherepn =d(-n+ 1)/2.

Proof. Pmv obviously satisfies part (a) of the theorem, while (b) follows by observing that A E Almvl implies An ::; m (with equality if and only if A = mv). •

(We refer the reader to [KS] for the precise sense in which this formula generalizes the classical Capelli identity. Also see [Sl, S2, BK] for some of the representation theoretic applications of this formula).

Our second application is an easy derivation of Wallach's formula [W] for Pvk. Let us define the symmetric polynomials a j and Tj of de­gree j, in n variables, via the generating functions n~l(1 + tXi) Ej=o tjaj(xt, ... ,xn) and n~l(1 + tXi)-l = E~o tjTj(Xt, .. · ,xn ).

Corollary 2. Pvk(x) equals E~=oTn-i(O, ... ,O,Pk+1"",Pn)ai(x),

Proof. The proposed polynomial is the coefficient of t k in the power series expansion of n~=l (1 + tXi)/ n~=k+l (1 + tPi), and clearly satisfies part· (a) of the theorem.

Now if A E Alvkl, A i- v k , then the last n - k coordinates of A are zero. For such A, if we put Xi = Ai + Pi, then the above power series be­comes a polynomial in t (of degree less than k), which implies the vanishing requirements of part (b). •

Except when n = 2, there does not seem to a nice closed formula for the general PI-'" However, in Section 2, we describe an inductive procedure for calculating these polynomials.

The proofs are almost embarrassingly easy.

O. Preliminaries

0.1 Hermitian symmetric spaces. Let us choose a Cartan subalge­bra ~ of t and 9 and fix positive root systems such that E+(~, 9) = E+(~,t) u E(~,p+). Let {'Yl,''','Yn} be the Harish-Chandra strongly orthogonal noncompact roots, and let t be the subalgebra of ~ spanned by their coroots.

Invariant Differential Opemtors 571

The weights of t in g form a root system of type en or Ben, and the (restricted) roots are {±(1'i ±1'j)/2,±1'i}, and possibly {±1'i/2}. The number d, referred to in the introduction, is the (common) multiplicity of the roots {±(1'i ± 1'j)/2}.

(For these and other standard facts about Hermitian symmetric spaces see [HI]).

As shown by Kostant-Joseph [J], and Schmid [Sc], the K-submodules of P occur with multiplicity 1, and their highest weights are of the form >'11'1 + ... + >'n1'n, where>. = (>'1,"', >'n) E A. For each such >., the corresponding submodule P>.. occurs in polynomials of degree 1>'1.

Since V is isomorphic to P*, we have compatible decompositions P = (J)>"EAP>.. and V = (J)>"EAV>.., where V>.. ~ P!. For I-' E A, choose a basis {cpj} ofPJ.I. and let {dj } be the dual basis ofVw Then DJ.I. = L.j CPj ®dj is an invariant differential operator of order 11-'1, which is independent of this choice. Moreover, the operators {DJ.I. I I-' E A} form a (vector space) basis for I. By Schur's Lemma, each DJ.I. preserves the decomposition of P, and acts by a scalar cJ.l.(>') on each P>...

This completes the explanation of the undefined terms in the introduc­tion.

0.2 Symmetric tube domains. The results of this section are due to Wallach [W].

It is well known that the Hermitian symmetric space G / K admits a realization as a generalized half plane (tube type domain) precisely when the root system E(t,g) is of type en (i.e. if there are no roots of the form ±'Yi/2).

Now suppose (g, t) is a Hermitian symmetric pair, not necessarily of tube type, and let 9 be the subalgebra generated by the root-spaces for {±(-Yi ± 1'j)/2, ±'Yi}. If we put t = 9 nt, and p± = 9 n I'±, then (g,t) is a Hermitian symmetric pair of tube type, and we have the decomposition 9 = t + p+ + p_.

(For example, if g = u(m, n) with m ~ n, then 9 = u(n, n». Let t be the subspace of p_ spanned by the root spaces for {-1'i/2},

then we have the dual decompositions p_ = p_ + t and p: = P: + t*.

Consequently, we get P = P + t*P, where P is the polynomial algebra on p_.

Now the t types of P are parametrized by the same set A as the t-types of P. Fix>. E A and let CP>.. be a highest weight vector in P>...

Lemma A. CP>.. belongs to P, and thus is a highest weight vector in P>...

Proof. Choose a basis of p_ consisting of t-weight vectors, and write CP>.. as a sum of monomials in the dual basis of p: = P: + t*. Then each of

572 S. Sahi

these monomials must have weight>. and degree 1>.1. For 0: = ant + ... + an'Yn E t*, let us define the total weight of 0: to

be at + ... + an. Then the basis vectors of p~ have total weight 1, while those of t* have total weight 1/2. So if a monomial of degree 1>'1 were to have one or more factors from t*, then its total weight would be less than 1>'1 and, in particular, its t-weight could not be >.. _

We now show that the problem of computing the eigenvalues can be reduced to the tube case. Proceeding as in the introduction, we define, for the pair (0', i), the invariant differential operators Dp. and their eigenvalues cp. (>.).

Lemma B. There is a nonzero scalar f\,p. such that c,..(>.) equals f\,,..c,..(>') for all >. E A.

Proof. For /1- E A we have the dual decompositions 'P p. = :p p. + 'P~ and

Vp. = i5p. + V~, where 'P~ C t*'P, and V~ C tV. From the definition of Dp. it follows easily that Dp. is a nonzero scalar multiple of Dp. + D~, where

D~ E 'P~ ®V~. If <P>. is as above, then Dp.<p>. = Cp.(>')<p>. and D,..<p>. = Cp.(>')<p>.. By

Lemma A, vector fields from t annihilate <P>.; thus D~<p>. = 0 and the lemma follows easily.

1. The combinatorial characterization

We are now ready for the proof of Theorem 1. In view of Lemma B, we may assume that (g, t) is of tube type. The chief advantage of this assumption is the following:

Consider the Harish-Chandra imbedding of G / K as a bounded domain in p_, and let 6 be its Shilov boundary. If G / K is of tube type then 6 is a symmetric space K / M for the action of K.

Moreover, we can choose M such that if t = m + 5 is the corresponding Cartan decomposition of t, then t (see section 0) is a maximal Cartan subspace of 5. The roots of tin e are bi - 'Yj)/2, each with multiplicity d. Thus the half sum of positive roots is E Pi'Yi where Pi = d( n - 2i + 1) /2 as in the introduction.

Now holomorphic functions on p_ are determined by their restrictions to 6, and, conversely, real analytic functions on 6 can be uniquely ex­tended to holomorphic functions in a neighborhood of 6. Thus if D is a holomorphic differential operator on p_, it has a unique restriction to 6 which satisfies DI6fl6 = (Df)16 for all functions f holomorphic in a neighborhood of 6.

Invariant Differential Opemtors 573

Proof of Theorem 1. As noted above, we assume that G I K is of tube type. Since D,. E I, its restriction D,.16 is an invariant differential operator of order IILI on the symmetric space KIM. By Theorem II.5.1S of [H2], we conclude that C,.(A) = p,.(A + p) where p,. is a polynomial of degree IILI on t* which is invariant under the Weyl group of E(t, t). Since this Weyl group is simply the symmetric group acting on the basis {'Yl, ... , 'Yn}, we get part (a) of the theorem.

The action of "differentiation" gives us a K -map from V ® P to P, where the image of V,. ®P>.. is contained in polynomials of degree IAI-IILI. Thus this image is zero if IAI < IILI; while if IAI = IILI, then the image is contained in e (the polynomials of degree zero), and thus we get a K­invariant pairing between P>.. and V,.. It follows easily that in the latter ca.<;e the image is nonzero if and only if A = IL.

Recalling the definition of D,., and applying these observations, we conclude that p,. satisfies part (b). It remains only to show that parts (a) and (b) characterize p,..

Let 8,,., be the space of symmetric polynomials (in n variables) of degree at most IILI. Then, with the notation as in the introduction, the map E akvk ~ n a~k gives a bijection between the set A,,., and a basis of 8,,.,. Thus the dimension of 8,,., is the same as the cardinality N, say, of A,,.,.

We claim that every polynomial in 8,,.,, in particular p,., is determined by its values on the set {A + P I A E A,,.,}. To see this we argue as follows:

"Evaluation" at the points in this set gives a linear map Ev from 8,,., to eN. Applying this map to {p>.. I A E A,,.,}, we get an N x N matrix whose (A, A')-th entry is p>..(A' + p). If we arrange the rows and columns in order of increasing IAI, then (b) implies that the matrix is triangular, with nonzero diagonal entries.

It follows easily that Ev is bijective, completing the proof of the theo-rem. •

2. The inductive formula

We now describe an inductive procedure for calculating p,. which is valid in a more general combinatorial context. Let p = (PI.· .. , Pn) be an arbitrary decreasing sequence of real numbers (i. e. P need not correspond to a root system).

Theorem 2. For each IL E A there is a polynomial P~ in n variables, unique up to a scalar multiple, which satisfies:

(a) P~ is symmetric and has degree at most IILI.

574 S. Sahi

(b) ~(It + p) is nonzero, while p~(A + p) is zero for all other A E AII'I'

Once the existence of the {~} is established, the uniqueness follows from the bijectivity of the Ev map exactly as in the proof of Theorem 1. However, since these polynomials are not related to invariant differen­tial operators, we no longer have an a priori existence result. We shall provide an alternative argument which has the added advantage of being mostly constructive. We preface the proof with two simple lemmas and a definition.

Shifting Lemma. Let r be any real number. Suppose Theorem 2 holds for some (p, It), then it holds for (p', It) where p' = (Pl + r, ... , Pn + r).

Moreover,~' (Xl.'" ,xn) = ~(Xl - r,'" ,Xn - r).

Proof. It is easy to see that v:: satisfies the conditions of theorem. •

Factoring Lemma. Let m be any positive integer. Suppose Theorem 2 holds for (p, It), then it holds for (p, It') where It' = (ltl + m, ... ,ltn + m).

Moreover, p~,(x) = (n~=l n~':l(Xi - Pn - j + 1») ~(xl-m, .. · ,Xn -m).

Proof. The proposed polynomial is easily seen to be symmetric and of degree 11t'1. It remains only to check the vanishing conditions of part (b)

for>.' E AII"I' If the last coordinate of A' is less than m, then (x n - Pn - A~) is a factor

of ~" and so the polynomial vanishes at x = >.' + p. On the other hand, if A~ ~ m, then>.' = A + (m,··· ,m), where A E AII'I; and the vanishing results follow by applying part (b) to ~ (A). •

Definition. Suppose p is a symmetric polynomial in n - 1 variables, and k is a real number. We shall write Symmk(p) for the polynomial in n variables given by

where the inner sum runs over all j-tuples 1 ::; it < ... < ij ::; n. It is easy to check that Symmk(p) is symmetric, has the same degree

as p, and satisfies Symmk(p)(Xl + k,···, Xn-l + k, k) = P(Xl."" xn-d for all Xl.'" ,Xn-l.

We are now ready to prove Theorem 2.

Proof of Theorem 2. We proceed by induction on n. If n = 1 and p = r and It = m, then the polynomial is ~(x) = n;:o\x - r - j). We now

Invariant Differential Opemtors 575

assume the result for (n - 1) variables and describe the construction in n variables. In view of the two lemmas, it suffices to construct ~ for n-tuples P and I-' whose last coordinates are zero.

We shall construct, by induction on k, polynomials Pk which satisfy part (a) of the theorem, and also satisfy part (b) for those A in AlI'I whose last coordinate is at most k. The desired polynomial will then be PI where I is the greatest integer not exceeding II-'I/n.

For k = 0, the polynomial Po is obtained by applying Symmo to the symmetric polynomial p«pl> ... ,pn-l» of degree 11-'1 in (n - 1) variables. The

1-'1,. .. ,l-'n-l latter polynomial exists in view of the inductive hypothesis on the number of variables,and using the remarks following the definition of Symmk> it is easy to check that Po satisfies part (b) for A E AII'I with An = O.

Now given Pk-l with k ~ I, if qk is any symmetric polynomial of degree II-'I-nk, then Pk = Pk-l - n~=l n~;:~(Xi - j)qk still satisfies (a) and (b) for A E AII'I with An ~ k - 1. We claim that there is a symmetric polynomial hk of degree 11-'1- nk in n -1 variables, such that if qk = Symmk(hk) then Pk also satisfies (b) for A E AII'I with An = k.

Rewriting these requirements, we see that hk must satisfy

n-l k

= Pk-l(AI + PI."', An-l + Pn-l)/k! II II (Ai + Pi + j) i=1 j=1

for all (AI. ... ,An-d E Aj;.I_nk' where the sets A-and A;;. are defined just as A and Am, but for (n - 1) variables.

As remarked above, the inductive hypothesis implies the bijectivity of the Ev map in (n - 1) variables. Consequently, we can find a (unique) polynomial hk satisfying these requirements. This completes the proof of theorem. •

Our argument is constructive, except for the definition of the hk which involves inverting the Ev map. However, even this is not too bad, since, as noted in the proof of Theorem 1, the matrix of this map is triangular with respect to a natural basis.

Finally, observe that for each p, the polynomials {p~ I I-' E A} give a (vector space) basis for the space of symmetric polynomials in n vari­ables. It would be interesting to express ~ in terms of other bases such as {na~k}.

576 S. Sahi

References

[BKJ R. Brylinski and B. Kostant, Minimal representations of E 6 , E7 and Es and the generalized Capelli identity, Proc. Natl. Acad. Sci. USA 91 (1994), 2469-2472.

[HI J S. Helgason, Differential Geometry, Lie Groups, and Symmetric Spaces, Academic Press, New York 1978.

[H2J S. Helgason, Groups and Geometric Analysis, Academic Press, New York 1984.

[JJ A. Joseph, A preparation theorem for the prime spectrum of a semisimple Lie algebra, J. of Algebra 48 (1977), 241-289.

[KSJ B. Kostant and S. Sahi, The Capelli identity, tube domains and the generalized Laplace transform, Adv. Math. 81 (1991), 71-92.

[SIJ S. Sahi, The Capelli identity and unitary representations, Compos. Math. 81 (1992), 247-260.

[S2] S. Sahi, Unitary representations on the Shilov boundary of a sym­metric tube domain, in: Representations of Groups and Algebras, Contemp. Math. vol. 145, Am. Math. Soc., Providence 1993.

[ScJ W. Schmid, Die Randwerte Holomorpher Funktionen auf Hermitesch Symmetrischen Riiumen, Invent. Math. 9 (1969), 61-80.

[WJ N. Wallach, Polynomial differential operators associated with Hermi­tian symmetric spaces, in: Representation Theory of Lie Groups and Lie Algebras, World Scientific, 1992, pp. 76-94, Singapore.

Mathematics Department, Rutgers University, New Brunswick, NJ 08903

Received January 21, 1994

Compact Subvarieties III Flag Domains

Joseph A. Wolf

to Bert Kostant on the occasion of his sixty-fifth birthday

ABSTRACT. A real reductive Lie group Go acts on complex flag manifolds G/Q. where Q is a parabolic subgroup of the complexification G of Go. The open orbits D = Go(x) include the homogeneous complex manifolds of the form Go/Vo where Vo = Go n Q", is the centralizer of a torus; those are the Go-homogeneous pseudo-kahler manifolds. For an appropriate choice Ko of maximal compact subgroup of Go. the orbit Y = Ko(x) is a maximal compact complex submanifold of D. The "cycle space" Mv = {gY I 9 E G and gY C D}, space of maximal compact linear subvarieties of D. has a natural complex structure. Mv plays important roles in the theory of moduli of compact kahler manifolds and in automorphic cohomology theory. Here we sketch a brief exposition of this interesting mathematical topic.

O. Introduction

Some of the most interesting homogeneous spaces in algebraic and differ­ential geometry are of the form D = Go(x) ~ GolVo where Go is a real semisimple group acting naturally on a flag manifold X = G I Q. Here G is the complexification of Go and Q is a "parabolic subgroup" of G. The complex flag manifolds themselves are examples of these real group orbits D = Go(x), corresponding to the case where Go is a compact real form of G. Grassmann manifolds and the other hermitian symmetric spaces of compact type are complex flag manifolds. The class of real group orbits D = Go(x) c GIQ = X on complex flag manifolds includes the noncom­pact "real forms" of complex flag manifolds, such as bounded symmetric domains, and also moduli spaces for polarized Hodge structures. These homogeneous spaces also play important roles in algebraic topology, in har­monic analysis (specifically in the representation theory for semisimple Lie groups), and in related theories of automorphic cohomology.

The theory is especially rich in the case of "flag domains", the case of measurable open orbits D = Go(x) c X. In this note I'll sketch the structure of those open orbits. They are important in harmonic analysis and representation theory for semisimple Lie groups (see [12], [13], [14], [16] and [24]), for the corresponding automorphic function theory (see [7],

Research partially supported by N.S.F. Grant DMS 91 00578.

578 J. A. Wolf

[8], [9], [20], [21] and [25]), and for their intrinsic geometric interest. The connection with representation theory is the realization of discrete series (or limits of discrete series) representations of Go as the natural action of Go on cohomologies H8(Dj JE) of homogeneous holomorphic vector bundles JE ~ D. The connection with automorphic function theory, as well as other aspects of complex geometry and complex harmonic analysis, comes through a certain "linear cycle space" MD of compact subvarieties Y c D. If ¢> E HB (Dj JE) with s = dime Y then integration over compact subvari­eties carries ¢> to a section of a certain vector bundle over MD. This carries a number of analytic problems from cohomology to the more accessible set­ting of sections of vector bundles. That sort of consideration plays a key role in the proof of FrEkhet convergence of certain Poincare series Er ,*(¢».

It is a pleasure to indicate some aspects of the theory of flag manifolds and flag domains in a volume celebrating this milestone occasion for Prof. Bertram Kostant. For he introduced me to the theory of real semisimple Lie groups and guided my first explorations of the orbit theory.

In §1 we indicate the structure of parabolic subgroups Q in a complex semisimple Lie group G and the corresponding flag manifold X = G / Q. This material was developed by Jacques Tits [17] in the 1950's. In §§2 and 3 we indicate a few generalities on orbits Go(x) C X where Go is a real form of X, and I'll look at the case where Go(x) is open in X. In §4 we sketch the situation ([22], [23]) for the hermitian symmetric case. It is fairly concrete and is needed later in §8. Then §5 returns to the case of open orbits, specializing to the measurable case, the case where D = Go(x) carries a Go-invariant pseudo-kahler metric. In that case we say that D is a "flag domain." The material of §§2, 3 and 5 represents joint work with Bert Kostant in the 1960's and is published in [22]. In §6 I sketch the construction of an exhaustion function for a flag domain D and the corresponding vanishing theorem for cohomologies of coherent analytic sheaves :F ~ D. This construction, due to Wilfried Schmid and myself [15] and based on earlier work of Schmid [12], comes into the construction of representations of Go on cohomologies of Go-homogeneous holomorphic vector bundles over D ([12], [13], [16], [20], [21]). It also comes into analysis ([20], [26]) of the space MD of maximal compact subvarieties of D. The coset space structure of MD is described in §7, and in §8 I indicate my proof [26] that MD is a Stein manifold.

1. Parabolic subgroups

Let g be a complex reductive Lie algebra, ~ a Cartan subalgebra and !:l. = !:l.(g,~) the root system of g with respect to~. Choose a positive root system !:l. + = !:l. + (g, ~). Then !:l. is the disjoint union of !:l. + with !:l. - = -!:l. +. Let

Compact Subvarieties in Flag Domains 579

\}I denote the corresponding simple root system. Thus every root a E ~

has unique expression a = E"'E\I1 n",1/J, the n", are integers, every n", ;;; 0 if a E ~ +, and every n", ~ 0 if a E ~ - .

Every subset ~ E \}I defines a subalgebra qcI> = q;;n + q4, c 9 where

q4, =~ + EOEcI>r go with ~r ={a E ~ I a = E"'EcI> n",'f/;} , q; =E,8EcI>-n g,8 with q>n ={a E ~+ I a fj. ~r}, (1.1)

q;;n=E,8EcI>-n g,8 with ~-n={a E ~- I a fj. ~r}.

Thus q;;n is the nilradical of qcI> and q4, is a reductive (Levi) complement. The 21\111 subalgebras qcI> C 9 are the standard parabolic subalgebras of g. The subalgebra q; + q4, egis called the opposite of qcI> .

Note that we have set things up so that the nilradical q;;n is spanned by negative root spaces. When we go to complex flag manifolds, this will mean that the holomorphic tangent space q; is spanned by positive root spaces, so positive bundles will correspond to positive linear functionals on ~.

The extreme cases are q\l1 = g, the entire algebra, and q0 = ~+ E,8EA- g,8, a Borel subalgebra. More generally, the maximal solvable subalgebras of 9 are called Borel subalgebras and are Int(g)-conjugate to q0, where Int(g) denotes the group of inner automorphisms of g. A subalgebra q C 9 is called a parabolic subalgebra if it contains a Borel subalgebra, and every parabolic subalgebra of 9 is Int(g)-conjugate to exactly one standard parabolic subalgebra.

Let G be a reductive complex Lie group. In other words, its Lie algebra 9 is reductive. If q = qcI> egis a parabolic subalgebra, then the corresponding parabolic subgroup of G is its normalizer

Q = QcI> = Na(q) = {g E G I Ad(g)q = q}. (1.2)

Let GO denote the topological component of the identity in G. Then QnGo is connected, that is, Q n GO = QO, for every parabolic subgroup Q C G. This is a consequence of simple transitivity of the Weyl group W(g,~) on the collection of simple subsystems of ~(g, ~). On the other hand, if Ad(g) is an inner automorphism of 9 then Q meets the component gGo of G. But if Ad(g) is outer this depends on ~.

Let LeG be a complex Lie subgroup. If G is connected then the following conditions are equivalent:

(i) L is a parabolic subgroup of G, (ii) the Lie algebra I of L is a parabolic subalgebra of g, and

(iii) the complex homogeneous space G / L is compact.

580 J. A. Wolf

Under these conditions GIL is called a complex flag manifold. Note here that L is connected and contains a Cartan subgroup, so GIL is simply connected.

More generally, whenever G is a reductive complex Lie group and Q is a parabolic subgroup, we say that X = G I Q is a complex flag manifold. Each topological component of X is simply connected. We view X as a complex manifold and G as a group of biholomorphic transformations of X. Since Q is the G-normalizer of q we sometimes view X as the space of all Ad( G)-conjugates of q. When G is connected this means that we sometimes view X as the space of all Int(g)-conjugates of q. In other words we sometimes identify x E X with the isotropy algebra

qx: Lie algebra of Qx = {g E G I g(x) = x}. (1.3)

2. Real group orbits

Let Go be a reductive real Lie group, go its real Lie algebra, and 9 = go 18> C its complexified Lie algebra. We fix

q: parabolic subalgebra of 9 (2.1a)

such that

if 9 E Go then Ad(g)q is Int(g)-conjugate to q. (2.1b)

This last condition is automatic if Go is connected. Let G be any connected complex Lie group with Lie algebra g. Then Go acts on the complex flag manifold

X = GIQ: all Int(g)-conjugates of q (2.2)

where Q is the parabolic subgroup of G that is the analytic subgroup for q. It acts through its adjoint action on g. Since we will only be interested in the Go-orbits and their structure, it will be convenient to assume, and we will assume, that

G is connected, simply connected and semisimple,

Go eGis the analytic subgroup for go .

Thus Go is a real form of G.

(2.3)

Go has isotropy subgroup Go n Qx at x E X. That subgroup has Lie algebra gonqx' The latter can be described as a real form of qxnrqx where r denotes complex conjugation of G over Go, of 9 over go. If ~ egis a r-stable Cartan subalgebra, then r acts on the root system 1::1 = 1::1(g,~) by (ro:)(e) = 0:(r-1e) = o:(re) for 0: E 1::1 and e E ~. Since the intersection of two Borel subalgebras contains a Cartan subalgebra, these considerations lead to

Compact Subvarieties in Flag Domains 581

2.4. Theorem. q", n rq", contains a r-stable Cartan subalgebra ~ of g, and there is a choice of positive root system 6.+ (g, ~) and subset cJ> c 'IT of simple roots such that q", = q~. That choice made, q", n rq", is the semidirect sum of its nilradical

{q", n rq",)-n = {q~ n rq~ + (q;n n rq~) + (q;n n rq;n)

E gfj + E gfj + E g'1

with its reductive (Levi) complement

(q", n rq", t = q~ n rq~ = ~ + E go<. ~rnT~r

In particular

(2.5a)

(2.5b)

(2.5c)

2.6. Corollary. The orbit Go(x) has real codimension 1cJ>-n n rcJ>-nl in X. In particular Go(x) is open in X if and only if cJ>-n n rcJ>-n is empty.

Each Go-orbit comes from a choice of Go-conjugacy class of pairs (Ho,6.+(g,~» where Ho is a Cartan subgroup of Go and 6.+{g,~) is a positive root system. The positive root system is determined only up to conjugacy within the Go-normalizer of Ho and the Weyl group W(q~, ~). Thus

2.7. Corollary. Let {H1,o, ... , Hm,o} be a complete system of conjugacy classes of Cartan subgroups of Go. Let Hj denote the Cartan subgroup ofG corresponding to ~j = ~j,O®C. Let Qj be a G-conjugate ofQ that contains Hj . In general let W(·,·) denote the corresponding Weyl group. Then there are at most

E IW(Go, Hj,o)\W(G,Hj)/W(Qj,H)1 (2.8) l~j~m

distinct Go-orbits on X. In particular, the number of Go-orbits on X is finite, so there are closed Go-orbits and there are open Go-orbits.

Here the closed orbit turns out to be unique. There is a good structure theory for the closed orbits, but it has not yet been exploited. We turn to the structure of the open orbits.

582 J. A. Wolf

3. Open orbits

Fix a Cartan involution {} of go and Go. In other words {} is an automor­phism of square 1 and, using (2.3) so that go is semisimple and Go has finite center, the fixed point set Ko = Gg is a maximal compact subgroup of Go. Thus go = to + So where to is the Lie algebra of Ko and is the ( + 1 )-eigenspace of {} on go, So is the (-1 )-eigenspace, to ..L So under the Killing form of go , and that Killing form is negative definite on to , positive definite on .so .

Every Cartan subalgebra of go is Ad(Go)-conjugate to a {}-stable Cartan subalgebra. A {}-stable Cartan subalgebra ~o c go is called fundamental if it maximizes dim (~o n to), compact if it is contained in to, which is a more stringent condition. More generally, a Cartan subalgebra of go is called fundamental if it is conjugate to a {}-stable fundamental Cartan subalgebra.

3.1. Lemma. The following conditions on a {}-stable Carian subalgebra ~o C go are equivalent:

(i) ~o is a fundamental Cartan subalgebra of go , (ii) ~o n to contains a regular element of go, and

(iii) there is a positive root system D. + = D. +(g, ~), ~ = ~o ®C, such that 7"D.+ = D.-.

A (}-stable Carian subalgebra ~o c go is compact if and only if 7" D. + = D.­for every positive root system D.+(g,~).

3.2. Theorem. Let X = G/Q be a complex flag manifold, G semisimple and simply connected, and let Go be a real form of G. The orbit Go(x) is open in X if and only if q", = q4> where

(i) ~o C q", n go is a fundamental Cartan subgroup of go and (ii) <P is a set of simple roots for a positive root system D. +(g, ~), ~ =

~o ® C, such that 7"D. + = D. -.

Fix ~o = (}~o, D. +(g,~) and <P as above. Let W(g, ~)f)o and W(q~, ~)f)o de­note the respective subgroups of Weyl groups that stabilize ~o. Then the open Go -orbits on X are parameterized by the double coset space W(t, ~ n t)\ W(g, ~)"o /W(q"" ~)f)o.

3.3. Corollary. Suppose that Go has a compact Carian subgroup, i. e. that to contains a Carian subalgebra of go. Then an orbit Go(x) is open in X if and only if go n qx contains a compact Carian subalgebra ~o of go , and then, in the notation of Theorem 3.2, the open Go-orbits on X are parameterized by W(t, ~)\W(g, ~)/W(q~, ~).

Classical methods of differential equations tell us when a homogeneous space carries an invariant complex structure [5]. The connection with

Compact Subvarieties in Flag Domains 583

semisimple structure theory was first sketched in the late 1950's ([2]. [3]), was connected to the theory of parabolic subgroups in the early 1960's [18], and was polished to an enumeration of invariant complex structures a few years later ([27], [28]). We apply this to Ko(x) C Go(x) C X. A careful examination of the way to sits in both t and go gives us

3.4. Theorem. Let X = G/Q be a complex flag manifold, G semisimple and simply connected, and let Go be a real form of G. Let x E X such that Go(x) is open in X, and let ~o C go n qx be a (}-stable fundamental Cartan subalgebra of go. Then Ko(x) is a compact complex submanifold of Go(x). Let K be the complexijication of K o , analytic subgroup of G with Lie algebra t = to ® <C. Then Ko(x) = K(x) ~ K/(K n Qx), complex flag manifold of K.

The compact subvariety Ko(x) controls the topology of an open orbit Go(x) C X, as follows. It follows from Corollary 3.3 that the compact real form aU. eGis transitive on X. That gives us a realization X = Gu/Vu where Vu C Gu is the centralizer there of a torus subgroup. In particular Vu is connected. In view of (2.3) now X is compact and simply connected. In view of Theorem 3.4, one can apply this argument to the compact subvariety Ko(x) C Go(x); so it is simply connected. Now a deformation argument shows that the open orbit Go(x) C X has Ko(x) as a deformation retract, so Go(x) is simply connected. Thus one obtains

3.5. Proposition. Let X = G /Q be a complex flag manifold, G semisim­pie and simply connected, and let Go be a real form of G. Let x E X such that Go(x) is open in X. Then Go(x) is simply connected and Go has connected isotropy subgroup (Qx n TQx)O at x.

The compact subvariety Y = Ko(x) also has a strong influence on the function theory for an open orbit D = Go(x) c X. The idea is that a holomorphic function on D must be constant on gY whenever g E G and g Y CD, so if there are "too many" translates of Y inside D then that holomorphic function must be constant on D. But this has to be formulated carefully.

Let X = G/Q be a complex flag manifold, G semisimple and simply connected, and let Go be a real form of G. Let x E X such that Go(x) is open in X. Then there are decompositions G = Gl X ••• x Gm and Q = Ql X ••• x Qm with Qi = Q n Gi and each Gi simple. Consider the corresponding decompositions X = Xl X ••• X Xm with Xi = GdQi and x = (x!, ... ,xm), Go = Gl,ox" .xGm,o, Go(x) = Gl,o(xI)x·· . x Gm.O(xm) and Ko(x) = Kl,o(Xl) x ... x Km,o(xm). If

(i) Gi,o n (Qi)x; = «Qi)x; n T(Qi)x;)O is compact, thus contained in Ki,o,

584 J. A. Wolf

(ii) Gi,O/ Ki,o is an hermitian symmetric coset space, and (iii) Gi,O(Xi) -+ Gi,o/ Ki,o is holomorphic for one of the two invariant

complex structures on Gi,o/ Ki,o

then we set Li = Ki so Li,o = Ki,o. Otherwise we set Li = Gi so Li,o = Gi,o. Note that each Gi,o/ Li,o is a bounded symmetric domain, irreducible or reduced to a point. Set L = Lo x ... x Lm so Lo = L1,0 X ••• x Lm,o. Then we say that

D(Go, x) = Go/Lo = (Gl,0/L1,0) x··· x (Gm,o/Lm,o) (3.6)

is the bounded symmetric domain subordinate to Go(x). Now we can state a precise result for holomorphic functions on Go (x).

3.7. Theorem. Let X = G / Q be a complex flag manifold, G semisimple and simply connected, and let Go be a real form ofG. Let x E X with Go(x) is open in X. Let D(Go,x) be the bounded symmetric domain subordinate to Go(x). Then 7r : g(x) t-+ gLo is a holomorphic map of Go(x) onto D(Go,x), and the holomorphic functions on Go(x) are just the 1 = f· 7r

where f : D( Go, x) -+ C is holomorphic.

Thus, in most cases there are no nonconstant holomorphic functions on Go(x), but in fact this depends on some delicate structure.

4. Example: Hermitian symmetric spaces

In this section, X = Gu / Ko is an irreducible hermitian symmetric space of compact type. Thus X = G/Q where G is a connected simply connected complex simple Lie group with a real form Go c G of hermitian type, as follows. Fix a Cartan involution () of Go and the corresponding eigenspace decomposition go = to + 50 where to is the Lie algebra of the fixed point set K o = Gg. Then Gu eGis the compact real form of G that is the analytic subgroup for the compact real form gu = to + 5 u of g where Su = A So of g.

There is a compact Cartan subalgebra to C to of go. If Q E ~(g, t) then either go: C t and we say that the root Q is compact, or Do: C 5 and we say that Q is noncompact. There is a simple root system \11 = {'l/Jo, ... ,'l/Jm} such that 'l/Jo is noncompact and the other 'l/Ji are compact. FUrthermore, every noncompact positive root is of the form 'l/Jo + LIS is m ni'I/Ji with each integer ni ;:;; O. If there are two root lengths then the noncom pact roots are long; this is immediate from the classification. Thus

g = t + 5+ + 5- where t = t + L go: ,5+ = L go: , no=O no=1

and 5- = L go:. no=-l

(4.1)

Compact Subvarieties in Flag Domains 585

Here q = q{'h, ... ,.pm}, in other words

The Cartan subalgebras of Uo all are Ad(Go)-conjugate to one of the ~r,o given as follows. Let r = {/1, ... '/T} be a set of noncompact positive roots that is strongly orthogonal in the sense that

if 1 ~ i < j ~ r then none of ± /i ± /j is a root. (4.3)

Then each ubi] = [U .. )';, U-"Yi] + U-Yi + U--Yi ~ s((2, q, say with

[U-y" U--Yi] 3 h-Yi ~ (~ ~1)' U-Yi 3 e-Yi ~ (~ ~), U--Yi 3 f-Yi ~ (~ ~),

such that UO[/i] = Uo n U-Yi ~ su(l, 1) is spanned by A h-yp e-Yi + f-Yi and A (e-Yi - f-yJ· Thus A h-Yi spans the compact Cartan subalgebra Ly; = UO[/i] n t of Uobi] and e-Yi + f-Yi spans the noncompact Cartan subalgebra a-Yi = UO[/i] ns of Uobi]. Strong orthogonality (4.3) says [U-ypU-YJ] = 0 for 1 ~ i < j ~ r. Define

tr = 2: Lyi and ar = 2: a-Yi' (4.4)

Then U has Cartan subalgebras

t=fr+(tntf) and ~r=ar+(tntf) (4.5)

They are Int(u)-conjugate, for the partial Cayley transform

Cr = II exp (~R (e-Yi - f-Yi») satisfies Ad(cr)tr = ar· (4.6) l~i~T

However, their real forms

to = Uo n t and ~r,o = Uo n ~r (4.7)

are not Ad(Go)-conjugate except in the trivial case where r is empty, for the Killing form has rank m = dim to and signature 21rl- m on ~r,o. More precisely,

586 J. A. Wolf

4.8. Proposition. Every Canan subalgebra of go is Ad(Go)-conjugate to one of the ~r,o, and Canan subalgebras ~r,o and ~r',o are Ad(Go)-conjugate if and only if the cardinalities Irl = Ir'l.

We recall Kostant's "cascade construction" of a maximal set of strongly orthogonal noncompact positive roots in b.(g, t). This set has cardinality f = rankRgo and is given by

3 ={<6, ... ,(il, where

<6 is the maximal (necessarily noncompact positive) root and (4.9)

(m+1 is a maximal noncompact positive root 1. {(l , ... , (m}.

Any set of strongly orthogonal noncompact positive roots in b.(g, t) is W(Go, To)-conjugate to a subset of 3. Further, the Weyl group W(Go, To) induces every permutation of 3.

Let Xo = 1· Q E G / Q = X, the base point of our flag manifold X when X is viewed as a homogeneous space. The Cartan subalgebra ~r,o c go leads to the orbits Go(cr4xo) c X where ruE is a set of strongly orthogonal noncompact positive roots in b.(g, t) with rand E disjoint. In view of the Weyl group equivalence just discussed, we may take r = {<6 , ... , (r}

and E = {(r+l,'" , (r+s}, both inside 3. Using Go = Koexp(as,o)Ko one arrives at

4.10. Theorem. The Go-orbits on X are just the orbits Dr,E = Go(crc~xo) where rand E are disjoint subsets of 3. Two such orbits Dr,E = Dr' ,E' if and only if cardinalities Irl = jr'l and lEI = IE'I. An orbit Dr,E is open if and only if r is empty, closed if and only if (r, E) = (3,0). An orbit Dr' ,E'

is in the closure of Dr,E if and only if IE'I ~ lEI and IE u rl ~ IE' u r'l·

S. Measurable open orbits

There is a class of open orbits that currently are much better understood than the general case. That is the class of measurable open orbits -the open orbits D = Go{x) c X where D carries a Go-invariant measure. They are characterized [22, Theorem 6.3] by

S.l. Theorem. Let D = Go(x), open orbit in X. If D is measurable then its Go-invariant measure is induced by the volume form of a Go-invariant indefinite-kiihler metric. FUnher, the following conditions are equivalent, and D is measurable if and only if they hold.

(i) Go n Qx is the centralizer of a torus subgroup Z of Ko n Qx , (ii) qx n rqx is reductive,

(iii) qx n rqx = q~ , (iv) rq;n = q~.

Compact Subvarieties in Flag Domains 587

Under these circumstances, Oq = q where () is the Cartan involution of go with fixed point set to .

The conditions of Theorem 5.1 are automatic if Ko contains a Cartan subgroup of Go, that is, if rankKo = rank Go, in particular if Go n Qx is compact. Thus in particular the open orbits Go(c~xo) of §4 are measurable. They are also automatic if Q is a Borel subgroup of G. More generally, they are equivalent [22, Theorem 6.7] to the condition that rq be Int(g)­conjugate to the parabolic subalgebra of g that is opposite to q.

6. Exhaustion functions for flag domains

Bounded symmetric domains D c en are convex, and thus Stein, so coho­mologies Hk(Dj F) = 0 for k > 0 whenever F ---+ D is a coherent analytic sheaf. This is useful for dealing with holomorphic discrete series representa­tions. More generally for dealing with general discrete series representations and their analytic continuations one has

6.1. Theorem. Let X = G/Q be a complex flag manifold, G semisimple and simply connected, and let Go be a real form of G. Let D = Go(x) c X = G/Q be a measurable open orbit. Let Y = Ko(x), maximal compact subvariety of D, and let s = dime Y. Then D is (s + I)-complete in the sense of Andreotti-Grauert [1]. In particular, if F ---+ D is a coherent analytic sheaf then Hk(DjF) = 0 for k > s.

The special case where Q is a Borel subgroup is due to Schmid [12], and the general case is due to Schmid and myself [15]. The arguments are similar: one examines the Levy form of an exhaustion function constructed from canonical line bundles. In this section I'll indicate the proof.

Let lKx ---+ X and lKD = KxlD ---+ D denote the canonical line bundles. Consider the dual bundles

lLx = lK:X ---+ X and lLD = lKh ---+ D. (6.2)

They are the homogeneous holomorphic line bundle over X associated to the holomorphic character

eA : Qx ---+ e defined by eA(q) = trace Ad(q)lq; . (6.3)

Write D = Go/Yo where Yo = Go n Qx, real form of Q~. It is convenient to write V for the complexification Q~ of Vo. Then, following standard terminology, PG IV is half the sum of the roots that occur in q~ and ), = 2PGlv, Thus, if 0: E ~(g, ~),

588 J. A. Wolf

(a, A) = 0 and a E q,r, or (a, A) > 0 and a E q,n, or (a, A) < 0

and a E q,-n.

Now TA = -A. Decompose go = to+.6o under the Cart an involution with fixed point set to, thus decomposing the Cartan subalgebra ~o ego n qx as ~o = to + no with to = ~o n to and ao = ~o n.6o. Then A(no) = O.

View D = Go/Vo and X = Gu/Vo where Gu is the analytic subgroup of G for the compact real form gu = to + A.6o. Then eA is a unitary character on Vo. Now

in

lLx -- X = Gu/Vo has a Gu-invariant hermitian metric hu,

lLD -- D = Go/Vo has a Go-invariant hermitian metric ho . (6.4)

We now have enough information to carry out a computation that results

6.5. Lemma. The hermitian form A 88hu on the holomorphic tangent bundle of X is negative definite. The hermitian form A 88ho on the holomorphic tangent bundle of D has signature n - 2s where n = dime D.

6.6. Corollary. Define ¢ : D -- JR by ¢ = log(ho/hu). Then the Levy form £(¢) has at least n - s positive eigenvalues at every point of D.

The next point is to show that ¢ is an exhaustion function for D, in other words that

{z E D I ¢(z) ~ c} is compact for every c E JR.

It suffices to show that e-cf> has a continuous extension from D to the compact manifold X that vanishes on the topological boundary bd(D) of Din X. For that, choose a Gu-invariant metric h~ on lLx = IKx normalized by huh~ = 1 on X, and a Go-invariant metric ho on lL v = IKD normalized by hoho = 1 on D. Then e-cf> = ho/h~. So it suffices to show that ho/h: has a continuous extension from D to X that vanishes on bd(D).

The holomorphic cotangent bundle Tx -- X has fibre Ad(g)(q;;)* = Ad(g)(q;n) at g(x). Thus its Gu-invariant hermitian metric is given on the fibre Ad(g)(q;n) at g(x) by Fu(e,'T}) = -(e,TO'T}) where (,) is the Killing form. Similarly the Go-invariant indefinite-hermitian metric on Tv -- D is given on the fibre Ad(g)(q;n) at g(x) by Fo(e, 'T}) = -(e, T'T}). But IKx = det Tx and IKD = det Tv, so

ho/h~ = c· (determinant of Fo with respect to Fu)

Compact Subvarieties in Flag Domains 589

for some nonzero real constant c. This extends from D to a Coo function on X given by

f(g(x» = c· (detFoIAd(g)(q;n) relative to detFuIAd(g)(q;n»). (6.7)

It remains only to show that the function f of (6.7) vanishes on bd(D). If g(x) E bd(D) then Go(g(x)) is not open in X, so Ad(g)(qx)+rAd(g)(qx) i= g. Thus

go C Ad(g)(q;n) but g-o ¢. Ad(g)(qx) + rAd(g)(qx)

for some a E 6.(g, Ad(g)I)). If (3 E 6.(g,Ad(g)l)) with g/3 C Ad(g)(q;n) then Fo(go,g/3) = 0, so

f(g(x» = O. Thus ¢ is an exhaustion function for D in X. In view of Corollary 6.6 now D is (s + 1)-complete. Theorem 6.1 follows.

7. Coset structure of the cycle space

In this section, X = G /Q is a complex flag manifold, G semisimple, con­nected and simply connected, and Go is a real form of G. We fix an open orbit D = Go(x) eX = G/Q and assume that it is measurable. As before, Y = Ko(x), maximal compact subvariety of D, and we write n = dime X and s = dime Y.

The linear cycle space or the space of maximal compact linear subvarieties of D is, by definition,

MD = {gY I 9 E G and gY CD}. (7.1)

Since Y is compact and D is open in X, MD is open in

Mx = {gY I 9 E G} ~ G/L (7.2a)

where

L = {g E G I gY = Y}, closed complex subgroup of G. (7.2b)

We now look at the structure of the G-stabilizer L of the maximal com­pact linear subvariety Y in our open orbit D = Go(x) ~ Go/Vo. The starting point is the following lemma, which is obvious.

7.3. Lemma. The kernel of the action of L = {g E G I gY = Y} on Y is

(7.4) kEK kEKo

590 J. A. Wolf

and KE C L C KQx.

In general, G,Go,Q,X,D,K,Ko and Y break up as direct products according to any decomposition of go as a direct sum of ideals, equivalently any decomposition of Go as a direct product. Here we use our assumption that G be connected and simply connected. So, for purposes of determining the group L specified in (7.2) and just above, we may and do assume that Go and go are noncompact and simple. This is equivalent to the assumption that Gol Ko be an irreducible riemannian symmetric space of noncompact type.

We will say that Go is of hermitian type if the irreducible riemannian symmetric space Gol Ko carries the structure of an hermitian symmetric space.

As before, we write () for the Cartan involution of Go with fixed point set K o , for its holomorphic extension to G, and for its differential on go and gj and we denote the (}-eigenspace decomposition by g = t + 5. Our irreducibility assumption says, exactly, that the adjoint action of Ko on So = go n 5 is irreducible. Go is of hermitian type if and only if this action fails to be absolutely irreducible. Let S± = exp(5±) C G. Then Gol Ko is an open Go-orbit on GIQ where Q = KS_ as in §4.

As before we have the compact real form Gu C G, real analytic subgroup for gu = to + A 50, and Ko = Go n Gu . Ko is its own normalizer in Go, but its normalizer NGu (Ko) in Gu can have several components.

7.5. Proposition. Either Go is of hermitian type and L = KE = KS±, connected, or l L = KNGu(Ko) with identity component LO = K. In either case Go n L = Ko. In general, if Go n Qx is compact then L = K E and L is connected.

This Proposition is proved by running through cases, which are as follows.

(1) Go is of hermitian type with Qx C KS_. (2) Go is of hermitian type with Qx C KS+. (3) Go is of hermitian type with Qx ¢.. KS_ , Qx ¢.. KS+, and S_ C

Qx. (4) Go is of hermitian type with Qx ¢.. KS_ , Qx ¢.. KS+, and S+ C

Qx. (5) Go is of hermitian type with Qx ¢.. KS_ , Qx ¢.. KS+, S_ ¢.. Qx,

and S+ ¢.. Qx. (6) Go is not of hermitian type.

An important consequence of Proposition 7.5 is

1 This latter situation occurs both for Go of hermitian type and for Go not of hermitian type.

Compact Subvarieties in Flag Domains 591

7.6. Corollary. Either L is a parabolic subgroup KS± of G and Mx = G / L is a projective algebraic variety, or L is a reductive subgroup of G with identity component K and M x = G / L is an affine algebraic variety.

8. Holomorphic structure of the cycle space

Here I want to indicate the proof of

8.1. Theorem. Let D = Go(x) be a measurable open Go-orbit on a complex flag manifold X = G / Q. Then the linear cycle space M D is a Stein manifold.

Results of this sort were suggested by a vanishing theorem in Schmid's thesis [12] and by Griffiths' discussion [8] of moduli spaces for compact Kaehler manifolds. One case was worked out by R. O. Wells, Jr. using explicit matrix calculations [19]. Later Wells and I gave an argument for Theorem 8.1 in the case where GonQx is compact [20, Theorem 2.5.6], but there were problems with the combinatorics of the proof. I settled these problems in the general case of Theorem 8.1, as stated, in [26], and that's the argument that is indicated below.

Consider the first of the two cases of Corollary 7.6.

8.2. Theorem. Suppose that Mx is a projective algebraic variety. Then every open orbit D = Go(x) c X is measurable and MD is a bounded symmetric domain. In particular MD is a Stein manifold.

Here Go is of hermitian type and L = KS±, maximal parabolic subgroup. We can replace D.+ by its negative if necessary and assume L = KS_. Thus we are in Case 1 (when Vo = Go n Qx is compact) or in Case 3 (when Vo = Go n Qx is noncompact) of the proof of Proposition 7.5. Also, Mx = G / L is the standard complex realization of the compact hermitian symmetric space Gu / Ko. Denote

G{D} = {g E G I gY CD}. (8.3)

It is an open subset of G, and MD C Mx 9:! G / L consists of the cosets gL with g E G{D}. Evidently MD is stable under the action of Go. Thus

G{D} is a union of double cosets GogL with 9 E G. (8.4)

The proof of Theorem 8.2 consists of showing that only the identity double coset occurs in GD. That relies on the orbit structure of Go on Mx as described in §4.

The case where Vo = Go n Qx is compact is easy. With Vo compact, Q., eLand there is a holomorphic fibration 7r : X ~ Mx given by

592 J. A. Wolf

gQx ~ gL. Here 71'(D) is the bounded symmetric domain {gL I 9 EGo} and the gY, 9 E G, are the fibres of 71' : X -4 Mx. Thus MD is the bounded symmetric domain {gL I 9 EGo}.

Return to the general case, where Vo may be noncompact. The double cosets GogL of (S.4) are in one to one correspondence with the Go-orbits on Mx. Those orbits were described in Theorem 4.10 above, and we use the notation of §4.

G{D} is open in G and the map G -4 G/L = Mx is open. So G{D}(z) is open in Mx. Thus, if cr/c~, E G{D}, and if Go(cr/c~/z) is in the closure of Go(crc~z), then crc~ E G{D}. Now (S.4) and Theorem 4.10 combine as follows.

8.5. Lemma. There is a (necessarily finite) set C of transforms crc~ , where rand E are disjoint subsets of 3, such that (i) if crc~, Cr'C~, E C with Irl = Ir'l and lEI = IE'I then r = r' and E = E' and (ii) G{D} = UCEC GocL. So if crc~ E C then CfUE' E C for every subset E' c E. In particular, if 4 f/. C whenever 0 =1= E c \[I then C = {I} and G{D} = GoL.

This reduces the proof of Theorem S.2 to the assertion 0 =1= E C 3 implies c~ ¢ C.

R = L n Qx is a parabolic subgroup of G and W = G/R is a complex flag manifold because S_ C L n Qx. So there are holomorphic projections

71" : W -4 X by gR ~ gQx , fibre F' = 71',-1 (x) = V(w) ~ V/V n L ,

71''' : W -4 Mx by gR ~ gL , fibre F" = 71',,-l(Z) = K(w) ~ K/ K n Qx , (S.6)

where 9 E G and w = 1· R in W. Set i5 = Go(w). Then

71": i5 -4 D by g(w) ~ g(x), fibre Vo(w) ~ Vo/Ko n Vo, open in F' . (S.7)

F' is a complex flag manifold of V = Q~, VO (w) is open in F', and Vo n K 0

is a maximal compact subgroup of Vo; so Vo(w) is a bounded symmetric domain and F' is its compact dual.

The usual positive definite hermitian inner product on go is (e, ry) = -bee, TOry) where b is the Killing form. The associated length function defines

lIello : operator norm of ad(e) : g -4 9 for e E g. (S.Sa)

The Hermann Convexity Theorem says

Go(z) = 71'''(D) = {exp(()(z) I ( E s+ with lIelio < I}. (S.Sb)

Recall tl = q~. A glance at the proof of the Harish-Chandra realization of Go(z) as a bounded symmetric domain, and of (S.Sb), shows that every

Compact Subvarieties in Flag Domains 593

9 E Go has expression

9 = exp(l + (2) . k· exp(17) where

17 E L, k E K, (2 E Ad(k)(u ns+), and (1 E s+..l Ad(k)(u ns+). (8.9)

There is a number a = aa > 0 such that, in (8.9), II(ll1g < aa. The operator norm information pulls back from G(z) to D. The result

is

8.10. Lemma. Decompose 9 E Go as in (8.9). Define 1: Go ~ 1R by 1(g) = II(ll1g· Then f(gx) = 1(g) is a well defined function f : D ~ lR. If gx E D then 0 ~ f(gx) < aa where aa is given as above.

Now the proof of Theorem 8.2 proceeds as follows. Let 0 i= E c S with c~ E C. In the notation of §4, we conjugate by an appropriate element of Ko and may assume E = {6, ... '~m} c S, with 1 ~ m ~ t. Let g[E] = L15 i5 m g[~i] and let G[E] be the corresponding analytic subgroup

of G. Then G{D} contains the diagonal subgroup Gd[E] ~ SL(2;C) in G[E]. Since 6 is not a root of q~ = u the orbit XdIE] = Gd[E](x) is a Riemann sphere contained in D.

Let E' = {a EEl gO" ¢. u = q~}, nonempty because it contains 6. The diagonal subgroup Gd[E'] ~ SL(2; C) in G[E'] has properties:

Gd[E'](x) = Xd[E'] is the same Riemann sphere Xd[E] ,

(gd[E'] ns+)..l (u ns+) , and

Gd[E'] n K is contained in the Cartan subgroup H with Lie algebra ~. (8.11)

Now look at the corresponding orbits in the hermitian symmetric flag variety Mx = G/KS_. The orbit Gd[E'](z), call it Zd[E'], is a Riemann sphere, the diagonal Z[E'] = G[E'](z). Its intersection with the bounded symmetric domain Go(z) is the hemisphere Gg[E'j(z), where Gg[E'] = Gon Gd[E'].

Let f* = llad[E/) in the notation of Lemma 8.10. Then f* is real analytic and has a unique real analytic extension ft to Gd[E'] nexp(s+)Kexp(.(L). Evidently ft is unbounded. The function f of Lemma 8.10 is real analytic on the lower hemisphere of the Riemann sphere Xd[E] = Xd[E'] and its restriction to that hemisphere has unique real analytic extension h to the complement Xd[E] \ c~(x) of the pole opposite to x, extension defined by

ft just as f is defined by 1 In view of (8.11), h = flxd[E)\4(x)' Since ft is unbounded, it follows that f is unbounded. This contradicts Lemma 8.10.

One concludes that C cannot contain any c~ with 0 i= E c S. As noted earlier, that completes the proof of Theorem 8.2.

The second case of Corollary 7.6 is addressed by

594 J. A. Wolf

8.12. Theorem. Suppose that the open orbit D C X is measurable and that Mx is an affine algebraic variety. Then MD is an open Stein subdo­main of the Stein manifold Mx.

Recall the exhaustion function ¢: D -+lR defined in Corollary 6.6. It is real analytic and its Levy form

(8.13)

has at least n - s positive eigenvalues at every point of D. Here n = dime D and s = dime Y. Since ¢ is an exhaustion function, the subdomains Dc = {z E D I ¢(z) < c} are relatively compact in D.

Analyticity allows one to transfer ¢ to MD. Define ¢M : MD -+lR+ by

¢M(gY) = sUPYEY ¢(g(y)) = SUPkEK¢(gk(x)) . (8.14)

The result is

8.15. Lemma. ¢M is a real analytic plurisubharmonic2 function on MD. [fYeo is a point on the boundary of MD in Mx and {Yi} is a sequence in MD that tends to Yeo then limy;->yoo ¢M(Yi) = 00.

The next step is to modify ¢ M to obtain a strictly plurisubharmonic exhaustion function on MD. Since Mx is affine, it is Stein, so there is a proper holomorphic embedding F : Mx -+ C2m+1 as a closed analytic submanifold of C2m+l. The norm square function N(m) = IIF(m)112 has positive definite Levy form, and the sets {m E Mx I N(m) < c} are relatively compact. Now

( : MD -+lR+ defined by (m) = ¢M(m) + N(m) (8.16)

has positive definite Levy form, thus is strictly plurisubharmonic. Since N and ¢ are real analytic, so is (. And (tends to 00 at every boundary point of MiJ because ¢ has that property by hypothesis and N has values ~ o. So every set

MC,c = {m E M I (m) < c} (8.17)

has closure contained in MD. But F is a proper embedding of Min C2m+l,

so the sets MC,c of (8.17) have compact closure in MD. We have proved that ( is a real analytic strictly plurisubharmonic ex­

haustion function on MD. Theorem 8.12 now follows using H. Grauert's solution [6] to the Levy Problem.

Note that this argument shows

2 A C2 function f on a complex manifold is called plurisubharmonic if the hermitian form C(f) is positive semidefinite at every point, strictly plurisubharmonic if CU) is positive definite everywhere. See [4], [11] or the exposition [10, §2.6].

Compact Subvarieties in Flag Domains 595

S.lS. Lemma. Let M be an open submanifold of a Stein manifold M. Suppose that M carries a C r plurisubharmonic function e , r E {2, 3, ... ,00, w} , that blows up on the boundary of M in M in the sense: if Yeo E bdM and {Yi} c M tends to Yeo then limi->eo e(Yi) = 00. Then M carries a Cr

strictly plurisubharmonic exhaustion function.

Theorem 8.1 follows from Theorem 8.2 when Mx is a projective alge­braic variety, from Theorem 8.12 when Mx is an affine algebraic variety. Corollary 7.6 says that these are the only cases.

REFERENCES

1. A. Andreotti and H. Grauert, Theoremes de finitude pour la cohomologie des espaces complexes, Bull. Soc. Math. France 90 (1962), 193-259.

2. A. Borel and F. Hirzebruch, Characteristic classes and homogeneous spaces, I, Amer. J. math. 30 (1958), 458-538.

3. A. Borel and F. Hirzebruch, Characteristic classes and homogeneous spaces, II, Amer. J. math. 31 (1959), 315-382.

4. H. Bremmermann, Complex convexity, Trans. Amer. Math. Soc. 82 (1956), 17-51. 5. A. Frolicher, Zur Differentialgeometrie der komplexen Strukturen, Math. Ann. 129

(1955), 5G-95. 6. H. Grauert, On LevY's problem and the imbedding of real-analytic manifolds, Ann.

of Math. 68 (1958), 46G-472. 7. P. A. Griffiths, Periods of integrals on algebraic manifolds, I, Amer. J. Math. 90

(1968), 568-626. 8. P. A. Griffiths, Periods of integrals on algebraic manifolds: Summary of main results

and discussion of open problems, Bull. Amer. Math. Soc. 76 (1970), 228-296. 9. P. A. Griffiths and W. Schmid, Locally homogeneous complex manifolds, Acta Math.

123 (1969), 253-302. 10. L. Hormander, An Introduction to Complex Analysis in Several Variables, Van Nos­

trand, 1966. 11. P. Lelong, La convexite et les fonctions analytiques de plusiers variables complexes,

J. Math. Pures et Appl. 31 (1952), 191-219. 12. W. Schmid, Homogeneous complex manifolds and representations of semisimple Lie

groups, thesis, University of California at Berkeley, 1967. 13. W. Schmid, On a conjecture of Langlands, Annals of Math. 93 (1971), 1-42. 14. W. Schmid, L2 cohomology and the discrete series, Annals of Math 103 (1976),

375-394. 15. W. Schmid and J. A. Wolf, A vanishing theorem for open orbits on complex flag

manifolds, Proc. Amer. Math. Soc. 92 (1984),461-464. 16. W. Schmid and J. A. Wolf, Geometric quantization and derived functor modules for

semisimple Lie groups, J. Funct. Anal. 90 (1990), 48-112. 17. J. Tits, Sur certains classes d'espaces homogenes de groupes de Lie, Memoir, Belgian

Academy of Sciences, 1955. 18. J. Tits, Espaces homogenes complexes compacts, Comment. Math. Helv. 37 (1962),

111-120. 19. R. O. Wells, Jr., Parameterizing the compact submanifolds of a period matrix domain

by a Stein manifold, in "Symposium on Several Complex Variables", Park City, Utah, 1970, Springer Lecture Notes in Math. 184, 121-150.

20. R. O. Wells, Jr., and J. A. Wolf, Poincare series and automorphic cohomology on flag domains, Annals of Math. (1977), 397-448.

596 J. A. Wolf

21. F. L. Williams, On the finiteness of the L2 automorphic cohomology of a flag do­main, J. Funct. Anal. 72 (1987),33-43.

22. J. A. Wolf, The action of a real semisimple group on a complex flag manifold, I: Orbit structure and holomorphic arc components, Bull. Amer. Math. Soc. 75 (1969), 1121-1237.

23. J. A. Wolf, Fine Structure of Hermitian Symmetric Spaces, in "Symmetric Spaces" , ed. W. M. Boothby and G. L. Weiss, Dekker, 1972,271-357.

24. J. A. Wolf, The Action of a Real Semisimple Group on a Complex Flag Manifold, II: Unitary Representations on Partially Holomorphic Cohomology Spaces, Memoirs Amer. Math. Soc., No. 138, 1974.

25. J. A. Wolf, Completeness of Poincare series for automorphic cohomology, Annals of Math. 109 (1979), 545-567.

26. J. A. Wolf, The Stein condition for cycle spaces of open orbits on complex flag manifolds, Annals of Math. 136 (1992), 541-555.

27. J. A. Wolf and A. Gray, Homogeneous spaces defined by Lie group automorphisms, I, J. Diff. Geometry 2 (1968), 77-114.

28. J. A. Wolf and A. Gray, Homogeneous spaces defined by Lie group automorphisms, II, J. Diff. Geometry 2 (1968), 115-159.

Department of Mathematics, University of California, Berkeley, California 94720,

Received April 4, 1994