UNIPOTENT REPRESENTATIONS ATTACHED TO SMALL NILPOTENT ORBITS

Jian-Shu Li

1. Introduction

This is about the construction and exhaustion of unipotent representations of classical groups attached to some small nilpotent coadjoint orbits. These examples of irreducible unitary representations will be isolated in the unitary dual and automorphic.

There are not many new results in this report. Anything that is new here comes from my joint work with Jing-Song Huang.

Przebinda has done a lot of work in this direction. Moeglin studied the p-adic version. Dual pairs in the stable range: D = ℝ, ℂ or ℍ, with usual involutions V, V′ = vector spaces over D, with non-degenerate sesqui-linear forms, one hermitian

and the other skew-hermitian.

G = U(V), G′ = U(V′) isometry groups. Then (G, G′) is a reductive dual pair in the symplectic group Sp(W), where W =

V ⊗_{D} V′, with (real) symplectic form <, >= tr_{D/ℝ}((·, ·) ⊗ \overline{(·, ·)′}).

The restriction of the Weil representation of Sp(W) ({ \atop Sp(W)}) then defines the theta correspndence between representations of G and G′.

Definition(Howe). We say (G, G′) is in the stable range with (say) G′ the smaller member if dim_{D}V′ ≤ the split rank of G (equivalently, the Witt index of V).

Example: (O(p, q), Sp_{2n}) is in the stable range if p, q ≥ 2n. Theorem. Let (G, G′) be a reductive dual pair in the stable range, with G′ the

smaller memeber. Then Howe duality gives rise to an injection

θ : Ĝ′ ↪ Ĝ

Typeset by AMS-TEX 2 JIAN-SHU LI

1991 Mathematics Subject Classification. Primary 22E45, 22E46.

Key words and phrases. nilpotent orbits, unipotent representations, rank of unitary representations, theta correspondence.

Partially supported by NSF grant no. DMS-9501092.

from the unitary dual of G′ to that of G.

Remarks:

1. Strictly, we should use two-fold covering groups of G and G′ and their genuine representations ....

2. This result essentially goes back to Howe. I made some small improvements and extensions.

3. θ is a topological isomorphism onto its image.

4. if dim V′ < split rank of G, then the image of θ is isolated from the rest of Ĝ. Howe suggested that one should be able to get an interesting set of isolated unipotent

representations using the map θ, starting from one dimensional representations:

Ĝ′_{1} ⟶ Ĝ_{1} ⟶ Ĝ′_{2} ⟶ ⋯ ⟶ Ĝ

This talk: try to substantiate this suggestion.

2. Characterization of the image of θ

Notion of rank: we define this in the case of the symplectic group. Let G = Sp_{2n} as above, P = MN its Siegel parabolic and N̂ ≃ Sym_{n}ℝ (the space of real symmetric n × n matrices view as an abelian group). For π ∈ Ĝ, π|_{N} is given by the spectral measure on N̂. The rank of π is the maximal rank of matrices in the support of π|_{N}.

We define the integer r_{G} as min(p, q) − 1 if G = O(p, q) and min(p, q) is odd. Otherwise r_{G} denotes the split rank of G.

Theorem. Let π be an irreducible unitary representation of G of rank < r_{G}. Then there exists a classical group G′ so that G, G′ form a reductive dual pair in the stable range, an irreducible representation σ of G′ and a one dimensional character χ of G such that π ≃ θ(σ) ⊗ χ.

These results essentially go back to Howe with some improvements by myself. (For the case of general linear groups see also Scaramuzzi’s thesis (Yale 1985).)

Let 𝔤_{0} = Lie(G) and 𝔤 = 𝔤_{0} ⊗ ℂ. We denote by W F (π) the wave front set (= asymptotic support of distribution characters) and by V(Ann(π)) the associated variety of the annihilator Ann(π) of π in U(𝔤).

Facts: 1. (Bohor-Brylinski) If π is irreducible then V(Ann(π)) = \overline{O} ⊆ 𝔤^{∗} is the closure of a single orbit.

2. W F (π) ⊆ 𝔤_{0}^{∗} ≃ 𝔤_{0} ⊆ 𝔤𝔩(V)

UNIPOTENT REPRESENTATIONS ATTACHED TO SMALL NILPOTENT ORBITS 3

a) W F (π) = AS(π) ≃ Vogan’s invariant.

b) (Barbasch-Vogan) W F (π) ⊆ \overline{O ∩ 𝔤_{0}^{∗}} Finally note W F (π) ⊆ GL(V). Definition:

rank (W F (π)) = max{rank x|x ∈ W F (π)}.

Theorem. Let π be an irreducible unitary representation of the classical group G. Then

rank π = min(rank of W F (π), r_{G})

Remark: 1. The fact that LHS ≥ RHS is due to Howe. 2. This result could serve as an alternative defintion of rank(π).

Corollary. Let π ∈ Ĝ with rank W F (π) < r_{G}. Then there exist G′, σ, χ with π = θ(σ) ⊗ χ.

3. Orbit correspondence via the moment map (real and complex versions)

Suppose that (G, G′) is a reductive dual pair in the stable range and π = θ(σ) where σ is an irreducible unitary representation of G′. How to relate W F (π) to W F (σ) and the associated varieties? The answer is again suggested by Howe. For the moment maps τ, τ′ : W ⟶ 𝔤_{0}^{∗}, 𝔤′_{0}^{∗} given by

τ(w)(x) =< x(w), w > (x ∈ 𝔤_{0}, w ∈ W)

we have 1. V(Ann(θ(σ))) = τ(τ′^{−1}(V(Ann(σ))).

2. W F (θ(σ)) = τ(τ′^{−1}(W F (σ)))

1. has been proven by Przebinda. Special cases of 2. was proven by Przebinda (under very severe additional conditions). We’ll treat 2. as a working Hypothesis. We write for the orbit correspondence O ↔ O′ if \overline{O} = τ(τ′^{−1}(\overline{O}′)).

Notation: O_{0} ⊆ 𝔤_{0}^{∗} , O ⊆ 𝔤^{∗}. Theorem. For dual pairs in the stable range, the orbit correspondence is given as

follows.

(a) Complex: [m_{1}^{r_{1}}, ⋯, m_{k}^{r_{k}}] ↦ [(m_{1} + 1)^{r_{1}}, ⋯, (m_{k} + 1)^{r_{k}}, 1^{∗}]

(b) Real: in terms of Young diagrams, as follows. Recall (see the book by Collinwood and McGovern) that a real orbit in a classical Lie algebra is given by a signed Young

4 JIAN-SHU LI

diagram with appropriate signature. Starting from the Young diagram of O we add a signed box on the right to each existing row; the sign is uniquely determined. Then we add one-box rows at the bottom until we reach the appropriate size and signature for a Young diagram representing a real orbit in 𝔤_{0}. This gives the orbit O.

Remark: Pan Shu-Yen has obtained this orbit correspondence in all cases (beyond the stable range).

Suppose O_{0} ↔ O′_{0}. We write O′_{0} = D(O_{0}).

Definition: (a) We say O_{0} is small if rank O_{0} < r_{G} and hence D(O_{0}) exists. (b) We say O_{0} is completely small if D^{j}(O_{0}) exists for j = 0, 1, 2 ⋯, until we get the zero orbit.

Example: Let G = U(p, q) with p, q sufficiently large. Then any rank 1 orbit is completely small. A rank 2 real orbit is small iff its Jordan blocks are given by [2, 2, 1, ⋯, 1]. (So any real orbit with Jordan blocks of the form [3, 1, ⋯, 1] is not completely small.)

Theorem. Let O_{0} be a completely small orbit. Then all irreducible unitary representations π with W F (π) = \overline{O}_{0} can be explicitly enumerated, starting from finite dimensionals and following through

Ĝ′_{1} ⟶ Ĝ_{1} ⟶ Ĝ′_{2} ⟶ ⋯ ⟶ Ĝ

.

Question: What happens for complex classical groups ?

Example: Take G = U(p, q).

(a) First let O_{0} = [3, 2_{+}, 2_{−}, 1^{∗}]. Here n_{+} denotes a n-box row in the Young diagram with + sign in the first box (and alternating signs down the row). Then D(O_{0}) = [2, 1_{+}, 1_{−}] of U(2, 2) and D^{2}(O_{0}) is the zero orbit for U(1). The representations in question are obtained by the chain of two maps

Û(1) ⟶ Û(2, 2) ⟶ Û(p, q)

(b) Now let O_{0} be any real form of [2^{4}, 1^{∗}]. Then D(O_{0}) is the zero orbit of U(r, s) with r + s = 4. The representations in question are obtained via

{finite dimensionals of U(r, s)} ⟶ Û(p, q)

Example: Take G = SO_{0}(p, q), G̃ = universal cover. Consider the complex orbit O = [m_{1}, m_{2}, ⋯, m_{r}, 1, ⋯, 1] with m_{j} > 1 for j = 1, ⋯, r and assume

m_{1} + ⋯ + m_{r} < 3r, m_{1} + ⋯ + m_{r} ≤ r + min(p, q) − 2

UNIPOTENT REPRESENTATIONS ATTACHED TO SMALL NILPOTENT ORBITS 5

(These are harsh conditions !)

Propposition. With the above assumptions, there exists no (unitary) representation π with V(Ann(π)) = \overline{O} unless m_{1} + ⋯ + m_{r} ≡ p + q (mod 2). In such a case π must factor through G.

Example of example: Take O = O_{m,k} = [3^{m}, 2^{2k}, 1^{∗}] with

k > 0, m + k ≤ \frac{1}{2}min(p, q) − 1

Then there exists no unitary representation π with V(Ann(π)) = \overline{O}_{m,k} unless m ≡ p + q (mod 2). If m ≡ p + q (mod 2) then all such representations are obtained via

\{finite dimensionals of O(r, s), r + s = m\} ⟶ { \atop Sp_{2(m+k)}} ⟶ Ĝ

Example of example of example: specializing to the case m = 0, k = 1, the above says that there exists no minimal representation of G̃ if p+q is odd and p, q ≥ 4. (Vogan + Howe). The result of Vogan is a little stronger in that he doesn’t require unitarity.

Example: Except for { \atop SO(3, 2n − 2)}_{0} and { \atop SO(2, 2n − 1)}_{0}, all minimal representations of classical groups are obtained by taking G′ as small as possible, and σ one dimensional.

Example: Sahi’s results (Inv. Math. 1992). Sahi dealt with P -orbits in 𝔫^{∗} rather than G-orbits in 𝔤^{∗}, construction but not exhaustion. What he constructed (excluding the E_{7} case) should coincide with some of the representations we treat here (case with σ = trivial of G′, one step).

References

Department of Mathematics, University of Maryland, College Park, MD 20742, U.S.A.

   E-mail address: jsl@math.umd.edu