UNITARY REPRESENTATIONS OF U(p, q)

PETER E. TRAPA

Abstract. Vogan has conjectured that a family of cohomologically induced modules exhaust the unitary dual of U(p, q) for certain kinds of integral infinitesimal characters. In this expository note (which is substantially a transcript of a talk given at the 1997 Seattle conference on the representation theory of real and p-adic groups) we recall the definition of the relevant modules, describe an approach to the conjecture, and discuss partial progress toward proving it.

1. Introduction

Let G be a real reductive Lie group with maximal compact subgroup K. The great unsolved problem of harmonic analysis is to understand the unitary dual, Ĝ_{u}, of G. By Ĝ_{u}, we mean infinitesimal equivalence classes of irreducible unitary (𝔤, K)-modules; here 𝔤 = Lie(G)_{ℂ}. The problem is notoriously difficult, and to approach it, one begins by studying a larger class of (𝔤, K)-modules containing Ĝ_{u}. The larger set, denoted Ĝ_{a}, consists of infinitesimal equivalence classes of irreducible admissible (𝔤, K)-modules, and is far more tractable. In particular, Ĝ_{a} can be explicitly parameterized in a number of ways. In any such parameterization, one begins by restricting to a large abelian subalgebra of the enveloping algebra U(𝔤). That restriction is codified in the notion of infinitesimal character which, for our purposes, we may take as a map

ic : Ĝ_{a} ⟶ 𝔥^{∗}/W;

here 𝔥 is a Cartan subalgebra of 𝔤 and W is the complex Weyl group W (𝔤, 𝔥).

Fix γ ∈ 𝔥^{∗}/W. (Sometimes we will think of γ as an orbit, other times as a particular representative.) Let Ĝ_{a}^{γ} denote the fiber of ic over γ, and similarly let Ĝ_{u}^{γ} = Ĝ_{a}^{γ} ∩ Ĝ_{u}. The character theory implies that Ĝ_{u}^{γ} ⊂ Ĝ_{a}^{γ} is an inclusion of finite sets, so that for each infinitesimal character, determining the unitary dual is a finite problem. But since the infinitesimal character is a purely algebraic invariant and has nothing to do with unitarity, we cannot expect the inclusion Ĝ_{u}^{γ} ⊂ Ĝ_{a}^{γ} to behave nicely with respect to γ. In fact, this is the case: as γ varies, essentially anything can happen. This at least begins to illustrate the subtlety of picking Ĝ_{u}^{γ} out of Ĝ_{a}^{γ}.

We will concentrate here on the particular example of U(p, q), the hope being that a complete understanding of examples will shed light on the problem in general. We will define U(p, q) below, but for now one need only know that it is a real form of 𝔤l(n, ℂ), n = p + q, so that an element γ ∈ 𝔥^{∗}/W is determined by an unordered n-tuple of complex numbers. Our interest is in a particular kind of integral γ; depending on the parity of n, we want to consider either tuples of integers or half-integers:

(1) γ ∈ \begin{cases}ℤ^{n} & if n is odd, \\ (\frac{2ℤ+1}{2})^{n} & if n is even. \end{cases}

The reason for this form of γ will become apparent below. For now, one should note that ρ (the half-sum of positive roots) is always of this form.

We are going to formulate a very precise conjecture, due to Vogan, about Ĝ_{u}^{γ} in the setting of the previous paragraph. But first we need to recall the (𝔤, K)-modules that enter its statement.

2. The weakly fair A_{𝔮}(λ) modules and a conjecture of Vogan

We will concentrate on our example explicitly, leaving it to the reader to supply the correspondence between our discussion and more general treatments (like the one given in [KV, Chapter 5]). As the name suggests, the A_{𝔮}(λ) modules depend on the parameters of 𝔮 and λ, which we now describe.

Fix positive integers p and q, and suppose {(p_{1}, q_{1}), …, (p_{r}, q_{r})} is an ordered sequence of pairs of positive integers with ∑_{i} p_{i} = p and ∑_{i} q_{i} = q. Set n = p + q and n_{i} = p_{i} + q_{i}. Define G = U(p, q) to be the set of n-by-n complex matrices that preserve a Hermitian form defined by a diagonal matrix whose first p_{1} entries are +1, whose next q_{1} entries are −1, whose next p_{2} entries are +1, and so on. Let 𝔤_{o} = 𝔲(p, q) be the corresponding Lie algebra so that 𝔤 = (𝔤_{o})_{ℂ} = 𝔤𝔩(n, ℂ). Next let T ⊂ G denote the diagonal (compact) Cartan, set 𝔱 = (𝔱_{o})_{ℂ}, and make the standard upper-triangular choice of positive roots. Let ρ denote half the sum of these positive roots.

Let L denote the subgroup of G consisting of block diagonal matrices whose block sizes are given by the n_{i}’s; then L ≅ ∏_{i} U(p_{i}, q_{i}). Let 𝔩_{o} denote its Lie algebra, and 𝔩 the corresponding complexification. Define 𝔮 = 𝔩⊕𝔲 to be the standard block upper triangular subgroup of 𝔤 with Levi subalgebra 𝔩. Let Q denote the corresponding subgroup of G_{ℂ} = GL(n, ℂ).

Next let ℂ_{λ} be a one dimensional unitary representation of L or (by complexifying and extending trivially on the nilradical) of Q. Then ℂ_{λ} has the form

det^{λ^{(1)}} ⊗ ⋯ ⊗ det^{λ^{(r)}}, λ^{(i)} ∈ ℤ.

For future reference, note that its differential restricted to 𝔱_{o} looks like

λ = (\overbrace{λ^{(1)}, …, λ^{(1)}}^{n_{1}}, …, …, \overbrace{λ^{(r)}, …, λ^{(r)}}^{n_{r}}). (2)

Form the Q-equivariant holomorphic line bundle

L′_{λ} = G_{ℂ} ×_{Q} (ℂ_{λ} ⊗ ⋀^{top} 𝔲)
↓
G_{ℂ}/Q.

(The appearance of ⋀^{top} 𝔲 is a convenient normalization which makes Theorem 2.3(a) work out nicely, for instance.) Now G acts on G_{ℂ}/Q on the left, and the orbit of eQ is noncompact, open, and diffeomorphic to G/L. Write i for the inclusion G/L ↪ G_{ℂ}/Q. Using it, we get a complex structure on G/L, and can pull back L′_{λ} to a holomorphic line bundle L_{λ} = i^{∗}L′_{λ} on G/L.

One can imagine λ controlling the geometry of L_{λ}. When λ is sufficiently positive (never mind exactly in what sense), we expect cohomology in only one degree. In the framework we have set up above, that degree of interest is S = dim_{ℂ}(𝔲∩𝔨). Here is the defintion of the (𝔤, K)-modules we have been leading up to.

Definition 2.1. The module A_{𝔮}(λ) is defined to be the underlying (𝔤, K)- module of the Sth Dolbeault cohomology group

H^{0,S}(G/L, L_{λ}).

There are a number of deep subtleties buried in this definition. The main point is that the Dolbeault cohomology groups carry no obviously nice topology, so there is nothing to guarantee that they are even representations of G. They are; but that has only been settled surprisingly recently ([Wo]). There is no harm in taking this result on faith, and proceeding naively.

To get a feel for the modules of the definition, let us consider some examples of them.

Example 2.2. 1. Suppose q = 0, so that G = U(n) is compact, and assume that λ + ρ is dominant and regular (or, explicitly by Equation (2), that λ^{(i)} −λ^{(i+1)} ≥ 0, for all i). Then the Borel-Weil-Bott theorem implies that A_{𝔮}(λ) is the unique irreducible representation of U(n) with highest weight λ.

2. Suppose each pair (p_{i}, q_{i}) is either (1, 0) or (0, 1); suppose further that λ + ρ is dominant and regular. Then L ≅ U(1)^{n} is a compact torus, and 𝔮 = 𝔟 is a Borel subalgebra of 𝔤. By Schmid’s thesis and its extensions, A_{𝔮}(λ) is the underlying (𝔤, K)-module of a discrete series representation with, roughly, Harish-Chandra parameter λ + ρ. (The last assertion is a little imprecise because, as we have set things up above, the definition of G depends on 𝔮.)

3. If 𝔮 = 𝔤 (that is, if our ordered sequence of pairs of integers consists of the single pair (p, q)), then A_{𝔮}(λ) = ℂ_{λ} is one dimensional.

4. In example 2, 𝔮 was as small as possible, while in 3, 𝔮 was maximal in size. In general, 𝔮 is somewhere in between, and the modules A_{𝔮}(λ) interpolate between the two extremes of discrete series and finite dimensional representations.

With these examples in mind, we can now state the properties of the A_{𝔮}(λ) modules. The first statement of the theorem is easy. The next one is much deeper; the unitarity part is due to Vogan, and for the range of λ given, so is the rest (though weaker versions go back to Zuckerman and others).

Theorem 2.3. Retain the notations given above. Then (a) A_{𝔮}(λ) has infinitesimal character λ + ρ.

(b) If

λ^{(i)} − λ^{(i+1)} ≥ −\frac{1}{2}(n_{i} + n_{i+1}),

then A_{𝔮}(λ) is unitarizable and either irreducible or zero.

There are several remarks to make about this theorem, but let us begin by considering the condition on λ. It arises sufficiently often that we set it aside in a definition.

Definition 2.4. If

λ^{(i)} − λ^{(i+1)} ≥ −\frac{1}{2}(n_{i} + n_{i+1}),

then λ is said to be in the weakly fair range for 𝔮, and the module A_{𝔮}(λ) is said to be weakly fair. (This definition can be easily seen to coincide with the more general coordinate free version given in [KV, Definition 0.52].)

Note that if the right hand side of condition in the defintion were increased to 0, then (by referring to Equation (2)) the inequality would imply that λ+ρ is dominant and regular. The relaxed condition of the weakly fair range thus allows the infinitesimal character λ + ρ to be substantially singular.

The next question to ask is whether or not Theorem 2.3 applies beyond the U(p, q) setting. One can certainly imagine the definition of A_{𝔮}(λ) being modified for arbitrary reductive G, and we have already given a reference to a general condition of weakly fair, so the theorem has an exact analog in general. Unfortunately, it is false: while the statements about infinitesimal character and unitarity still hold, the last assertion about being either irreducible or zero fails in general. The key feature of the U(p, q) case is that the structure of nilpotent coadjoint orbits in 𝔤l(n, ℂ) is very simple, the relevant fact being that they are all normal.

A final question is to ask how well understood the modules appearing in the theorem are. When λ + ρ is dominant and regular, Example 2.2.2 suggests that we may be able to rely on intuition based on the discrete series which, of course, are very well understood. This is essentially correct: when λ + ρ is dominant and regular, the Langlands parameters of the A_{𝔮}(λ) are relatively easy to describe (see [VZ, Section 6]). But outside this range the situation is considerably more complicated. Determining the Langlands parameters of a given weakly fair A_{𝔮}(λ) for U(p, q) (or even determining if it is nonzero) has only recently been solved; below, we will discuss this further. For general G, the issue of Langlands parameters is still open, though recent work of McGovern ([Mc1]) suggests that the cases of Sp(p, q) and SU^{∗}(2n) might be handled by the methods of the U(p, q) case.

In any event, as we vary the choice of the sequence {(p_{1}, q_{1}), … (p_{r}, q_{r})} (and hence of 𝔮) and as we vary λ, Theorem 2.3 provides a large catalog of irreducible, unitary (𝔤, K)-modules for U(p, q)^{1}. Conjecturally, the catalog is complete.

Conjecture 2.5 (Vogan). Suppose G = U(p, q) and γ is as in Equation (1). Then

Ĝ_{u}^{γ} = {A_{𝔮}(λ) | λ is weakly fair for 𝔮, and
λ + ρ ∈ W γ}.

The reason for the Equation (1) condition on γ is now clear: those γ are precisely the ones that arise as the infinitesimal character of an A_{𝔮}(λ). The conjecture says that for that kind of γ, all irreducible unitary (𝔤, K)-modules are A_{𝔮}(λ)’s. When γ is additionally assumed to be regular, Conjecture 2.5 is a theorem due to Salamanca. We shall return to this in the next section where we discuss approaches to Conjecture 2.5.

3. Approaches and partial progress toward conjecture 2.5

There is a fairly standard approach to proving a statement like Conjecture 2.5. First recall the inclusion Ĝ_{u}^{γ} ⊂ Ĝ_{a}^{γ} from Section 1. In our setting, the approach conists of three steps:

1. Parameterize Ĝ_{a}^{γ}.

2. Locate the parameters of the weakly fair A_{𝔮}(λ) with infinitesimal character γ.

3. Prove that all other parameters give rise to nonunitary (𝔤, K)-modules. For γ regular, this program was successfully completed by Salamanca in her thesis ([Sa1]), which has been elegantly simplified recently ([Sa2]) using the powerful ideas of Salamanca-Vogan ([SaV]). More precisely, in terms of the steps outlined above, the parameterization of Ĝ_{a}^{γ} is by Vogan’s theory of lowest K types; the identification is based on results of Vogan-Zuckerman ([VZ, Proposition 6.1]) depending on the regularity hypothesis; and the exhaustion step follows by a careful reduction argument (using the results of [SaV]) ultimately relying on Parthasarathy’s Dirac operator inequality.

1Recall that our definition of U(p, q) depends on the choice of {(p_{1}, q_{1}), …, (p_{r}, q_{r})}.

(This has been done to make the description of 𝔮 more transparent.) But the variously defined U(p, q)’s are all conjugate inside GL(n, ℂ) by the action of W, so this is a minor point.

Regardless of the details, one need only understand that Salamanca’s proof of Conjecture 2.5 for regular infinitesimal character depends heavily on that key hypothesis. To make progress on the singular case, we need to overhaul the entire approach, beginning with the parameterization of Ĝ_{a}^{γ}. The next theorem is such a parameterization; in its statement, many terms are undefined, but we shall define them in the remarks following the theorem.

Theorem 3.1 (Barbasch-Vogan [BV2]). Let G = U(p, q). The map

Ĝ_{a}^{γ} ⟶ Prim(U(𝔤))_{γ} × {nilpotent orbits in 𝔲(p, q)}

assigning to each X ∈ Ĝ_{a}^{γ} the pair (Ann(X), AS(X)) consisting of its annihilator and its asymptotic support is a bijection onto the same shape subset of the image.

Obviously there is a lot to sort out in the statement of this theorem. First, Prim(U(𝔤))_{γ} denotes the set of primitive ideals with infinitesimal character γ. Joseph has classified this set explicitly in terms of tableaux. More precisely, (when γ is real) he proved that Prim(U(𝔤))_{γ} is in bijection with the set of γ-standard tableaux. Here a γ-tableau consists of a Young diagram of size n filled with the coordinate entries of γ; the ‘standard’ condition amounts to requiring that the entries weakly decrease across rows and strictly decrease down columns.

The asymptotic support of an irreducible (𝔤, K) module is an important invariant originally defined by Barbasch-Vogan in [BV1]. Its precise definition is a little too complicated to recall here; roughly, it is a measure of the singularity of the corresponding distribution character at the identity. For an arbitrary X ∈ Ĝ_{a}^{γ}, AS(X) is a union of closures of real nilpotent orbits, so it is not at all clear that the map of Theorem 3.1 is even well defined. We will come back to this in a moment, but assume for now that AS(X) is (the closure of) a single nilpotent orbit. Now the set of nilpotent orbits in 𝔲(p, q) is in bijection with certain signed Young tableaux (see [CMc, Theorem 9.3.3]). Combining this with Joseph’s parameterization of Prim(U(𝔤))_{γ}, it is now clear what the ‘same shape subset’ of

Ĝ_{a}^{γ} ⟶ Prim(U(𝔤))_{γ} × {nilpotent orbits in 𝔲(p, q)}

means.

Note that given the explicit tableau-level descriptions of the same shape subset of the image, the theorem is truly a parameterization of Ĝ_{a}^{γ}: the statement describes a bijection between Ĝ_{a}^{γ} and the concrete set of sameshape pairs of certain kinds of tableaux.

Let us consider where the theorem comes from; along the way, we will see why the map of the theorem is well defined. For definiteness, fix γ = ρ; the general case follows by an appropriate translation principle. Barbasch and Vogan introduced an equivalence relation on the set Ĝ_{a}^{ρ}, the equivalence classes of which are called (Harish-Chandra) cells. Each cell carries the structure of a W -module. (In fact, the modules in each cell parameterize an integral basis for the minimal subquotients of the coherent continuation representation on the Grothendieck of formal characters spanned by those of Ĝ_{a}^{ρ}.) Inside each cell, Barbasch and Vogan located a canonically defined A_{𝔮}(λ) module whose asymptotic support they explicitly computed to be the closure of a single nilpotent orbit in 𝔲(p, q). Moreover, all nilpotent orbits arise in this way. For mostly formal reasons, the asymptotic support is constant on cells, thus implying that the asymptotic support of any member of Ĝ_{a}^{ρ} (and, by a translation principle, any Ĝ_{a}^{γ}) is the closure of a single nilpotent orbit. In particular, this shows that the map of the theorem is well defined.

Now the statement of the theorem at least makes sense, but since we have come this far, we might as well finish sketching its proof. So far, we have broken Ĝ_{a}^{ρ} into a disjoint union of cells parameterized by nilpotent orbits in 𝔲(p, q) or, equivalently, parameterized by certain signed Young tableaux of size n. As we have said, the elements of a given cell parameterize a basis for a representation of W ≅ S_{n}. Barbasch and Vogan computed that this representation is an irreducible representation of S_{n} and, in Young’s notation, is simply the one given by the Young diagram which is the shape of the signed Young tableau parameterizing the cell. The dimension of this representation is the number of Young tableaux of this shape; of course, this is also the number of ρ-standard tableaux. To finish proving the theorem, one needs to know that all elements of a given cell have distinct annihilators and that the shape of the annihilator and asymptotic support of a given element coincide. Both of these facts are reasonably well known (see [V1, Theorem 3.2] and [BV1, Theorem 4.1], respectively).

Theorem 3.1 completes the first step in the approach to Conjecture 2.5 described above. In this context, the first step is useful only if the (more difficult) second and third steps are tractable. Here is the second step:

Theorem 3.2 (Trapa [Tr]). Given a pair of tableaux (S, S_{±}) consisting of a γ-standard tableau of size n = p + q and a signature (p, q) signed tableau of the same shape, there is an algorithm to determine all weakly fair A_{𝔮}(λ) with

(Ann(A_{𝔮}(λ)), AS(A_{𝔮}(λ))) = (S, S_{±}).

Beyond noting that the theorem identifies the weakly fair A_{𝔮}(λ) in the Barbasch-Vogan parameterization, there are several useful remarks to make. The theorem follows by formally inverting an algorithm which, given a λ in the weakly fair range for a given 𝔮, computes the annihilator and asymptotic support of the corresponding A_{𝔮}(λ). Coupled with Garfinkle’s algorithm ([G]), this gives the Langlands parameters of any weakly fair A_{𝔮}(λ) for U(p, q):

Corollary 3.3. Given λ in the weakly fair range for 𝔮, there is an algorithm to determine the Langlands parameters of A_{𝔮}(λ).

A slight caveat is in order here: the Langlands parameters are obtained by an algorithmic procedure, so it is difficult to make general statements about them. But even so, the statements that are available have interesting applications in their own right.

Finally, there is the looming question of the third piece of the approach to Conjecture 2.5, the exhaustion step. In practice, the results of [SaV] (some of which are still partly conjectural) reduce the problem to a finite set of small infinitesimal characters. For any particular U(p, q), one can write down the finite set of tableau parameters for the Ĝ_{a}^{γ} with these infinitesimal characters, and using Theorem 3.2, one can cross off the parameters giving rise to the weakly fair A_{𝔮}(λ) modules. It remains to show that the remaining pairs of tableaux correspond to nonunitary (𝔤, K)-modules.

For a given U(p, q), this is a very concrete finite problem. Examples immediately indicate that current techniques are inadequate to handle it however. For instance, in U(3, 2) there is an irreducible admissible X (that is not a weakly fair A_{𝔮}(λ)) whose infinitesimal character is in the convex hull of ρ, but for which the Dirac operator inequality is inconclusive on all K types. One can in fact prove that X is nonunitary using an ad hoc argument. From this example emerges a means to attack the exhaustion step: one might hope that the new techniques developed to prove nonunitarity in examples become systematic enough to handle the general case.

4. Acknowledgements

As mentioned in the abstract, this article is based on a talk given at the joint AMS-IMS-SIAM conference on the representation theory of real and p-adic groups held in Seattle during July, 1997. I would like to thank the conference organizers for the opportunity to give this talk. Much of what appears here is a part of my MIT thesis, and it is a pleasure to thank my thesis advisor, David Vogan, for introducing me to this beautiful subject.

References

[BV1] Barbasch, D. and D. A. Vogan, Jr., The local structure of characters, J. of Func. Anal., 34(1980), 27–55.

[BV2] Barbasch, D. and D. A. Vogan, Jr., Weyl Group Representations and Nilpotent Orbits, Proceeding of the Park City Conference on Harmonic Analysis, P. Trombi ed., Progress in Math, Birkhäuser (Boston), 1982, pp 21–33.

[CMc] Collingwood, D. H. and W. M. McGovern, Nilpotent orbits in semisimple Lie algebras, Chapman and Hall (London), 1994.

[G] Garfinkle, D., The annihilators of irreducible Harish-Chandra modules for SU(p, q) and other type A_{n−1} groups, Amer. J. Math., 115(1993), 305–369.

[KV] Knapp, A. W. and D. A. Vogan, Jr., Cohomological Induction and Unitary Representations, Princeton Mathematical Series, number 35, Princeton University Press, 1995.

[Mc1] McGovern, W. M., Cells of Harish-Chandra modules for real classical Lie groups, to appear in Amer. J. Math.

[Sa1] Salamanca-Riba, S., The unitary duals of some classical Lie groups, Comp. Math., 68(1988) 251–303

[Sa2] Salamanca-Riba, S., On the unitary duals of real semisimple Lie groups and the A_{𝔮}(λ) modules: the regular case, preprint.

[SaV] Salamanca-Riba, S. and D. A. Vogan, Jr., On the classification of unitary representations of reductive Lie groups, preprint.

[Tr] Trapa, Peter E., Annihilators and associated varieties of the weakly fair A_{𝔮}(λ) modules for U(p, q), in preparation.

[V1] Vogan, D. A., Jr., A generalized τ-invariant for the primitive spectrum of a semisimple Lie algebra, Math Ann, 242(1977), 209–224.

[VZ] Vogan, D. A., Jr., and Gregg Zuckerman, Unitary Representations with nonzero cohomology, Comp. Math., 120 (1984) 141–187.

[Wo] Wong, H. W., Dolbeault cohomology and Zuckerman modules associated with finite rank representations, Harvard University Dissertation, 1992.

Department of Mathematics, M.I.T., Cambridge, MA 02139

E-mail address: ptrapa@math.mit.edu