ORBITAL SYMMETRIC SPACES AND FINITE MULTIPLICITY

Ronald L. Lipsman

Department of Mathematics University of Maryland College Park, Maryland 20742 USA

Abstract. It is proven that any algebraic symmetric space has finite multiplicity. The multiplicity is then shown to be bounded provided that is the case for reductive symmetric spaces (a widely-believed, but as yet unproven fact). The method of proof involves an amalgamation of the Mackey Machine and the Orbit Method. A necessary and sufficient orbital condition is also derived for the quasi-regular representation of a symmetric space to be irreducible.

1. Introduction. On several instances previously I have examined the harmonic analysis of non-semisimple symmetric spaces (e.g. in [11] [13, §6] [14] [16,§4] and [19]). One interesting category is that of abelian symmetric spaces — namely G/H, where G = H ⋉V is a semidirect product of a normal real vector subgroup V by any Lie group H acting on V satisfying the condition that V̂ /H is countably separated. Then the spectrum of the quasiregular representation τ = Ind_{H}^{G}1 is multiplicity-free and parameterized by V̂ /H. However, it is not at all difficult to find examples in which V̂ /H is measure-theoretically a singleton — e.g. when H has a Zariski-open orbit on V̂. In that case the representation τ is actually irreducible. At first glance that would seem almost pathological — a symmetric space whose associated quasi-regular representation is irreducible! One unavoidable consequence (we shall discover) is that the algebra of G-invariant differential operators D(G/H) is trivial. Hence, on a first encounter, it is entirely reasonable to view such spaces as unusual or atypical, and therefore not to concentrate attention too closely on them.

But surprisingly, as one broadens the category of non-semisimple symmetric spaces one investigates, this phenomenon keeps recurring. For example, in a natural generalization of abelian symmetric spaces — the so-called Strichartz symmetric spaces — we see it happening with great regularity (see sections 2 and 8 for examples). And although it does not happen for semisimple or nilpotent symmetric spaces, it does occur quite frequently for solvable symmetric spaces. If we only ask that τ be a finite sum of irreducibles — a situation which also appears unusual at first glance — then the occurrence of examples is even greater. Finally, if we write 𝔤 = 𝔥 + 𝔮 for the ±1-eigenspace decomposition of the Lie algebra (with respect to the differentiated involution), then typically (even though D(G/H) is trivial) the algebra of H-invariant differential operators on 𝔮 will not be trivial, if H is not reductive. For all these reasons, one is obliged to look more closely at these spaces.

Typeset by AMS-TEX

1991 Mathematics Subject Classification. 22E45.

Supported by NSF # DMS-90-02642 and # DMS-92-25-063

In fact, it is possible to give an orbital characterization of these “minimal” symmetric spaces. I shall do so in this paper. But interestingly, I shall show how the tools developed for arriving at that characterization provide me with a method for settling an important open conjecture concerning general symmetric spaces. What might be an orbital characterization of symmetric spaces with irreducible τ? If one consults the formulas in [12] or [15], it is clear that the phenomenon should demand that the space 𝔥^{⊥}/H of H-orbits on 𝔥^{⊥} be measure-theoretically a singleton. As in the abelian symmetric space example above, that suggests a Zariski-open orbit, and in fact I shall work in the context of real algebraic groups. Such a context is already strongly suggested by the structure of Strichartz symmetric spaces. But we must also deal with the fact that (outside exponential solvable groups) there is a second parameter — in addition to geometric (coadjoint) orbits — that appears in the orbital description of the irreducibles — namely, the dual of the component group of the stabilizer of functionals in the orbits. Taking that into consideration, I shall prove in this paper

Theorem 1.1. Let G be real algebraic, σ an involution of G, H = G^{σ}. Then the quasiregular representation τ = Ind_{H}^{G}1 is

(i) irreducible ⇔ H has a Zariski-open orbit on 𝔥^{⊥} and for any ϕ in that orbit, G_{ϕ} ⊂ H; (ii) a finite sum of irreducibles ⇔ there is a finite collection of open H-orbits in 𝔥^{⊥}

whose union is conull.

(Note that either condition in (i) requires that Cent G ⊂ H.) How can such a theorem be proven? Based on [8], it seems clear that a proof must involve a scheme which incorporates an amalgamation of the Mackey Machine and the Orbit Method. In attempting to set up such a scheme I noticed that the “little” homogeneous spaces that arise nearly reflect the nature of the original symmetric space G/H, but of course have smaller dimension. This suggests the possibility of an inductive procedure. In fact I can implement such a procedure (see §5). Moreover, in addition to the proof of Theorem 1.1, it enables me to settle the conjecture in [18] — namely I can prove

Theorem 1.2. Let G be real algebraic, σ an involution of G, H = G^{σ}. Then the quasiregular representation τ = Ind_{H}^{G}1 has finite multiplicity.

This theorem generalizes van den Ban’s theorem in the reductive case [2] and Benoist’s theorem in the exponential solvable case [4]. In the latter, the multiplicity is free. While this is not so in the reductive case, it is expected (although not proven yet apparently) that the multiplicity is bounded. I shall show that the multiplicity of any algebraic symmetric space is bounded if it is true in the reductive case (see Theorem 7.3 below).

One detail in the proof of Theorem 1.2 is worth mentioning at this point. One knows that typically one can expect cocycle obstructions to be associated with the little groups (or homogeneous spaces) that arise in the Mackey Machine. However, their description has been essentially integrated into the Orbit Method in [7]. Nevertheless, this appearance has the potential of disrupting the inductive scheme since the little homogeneous spaces may have a twist that does not appear in the original. One can overcome that problem by installing the twist into the original structure — the little homogeneous space twist will be of exactly the same type. We shall do this, and in fact we shall deduce Theorem 1.2 from the following more general theorem, which we shall prove by the enhanced inductive scheme.

Theorem 1.3. Let G be real algebraic, σ an involution of G, H = G^{σ}. Let Z ⊂ Cent G be a unipotent connected σ-invariant subgroup. Suppose χ is a unitary character of HZ satisfying σχ = χ̄. Then the induced representation

τ_{χ} = Ind_{HZ}^{G}χ

has finite multiplicity.

The organization of the paper is as follows. In section 2 we present some of the most elementary examples of symmetric spaces with τ irreducible or a finite sum of irreducibles. Motivated by Theorem 1.1, we call these symmetric spaces (or their quasi-regular representations τ) orbital or quasi-orbital respectively. Section 3 contains two auxiliary results:

(i) the fact that any involution preserves some Levi component; and (ii) the computation of the Penney distribution spaces for an orbital symmetric space. The latter establishes that the distribution-theoretic form of Frobenius reciprocity holds for orbital symmetric spaces. In section 4 we make the orbital terminology precise. Then in section 5 we develop in detail the inductive scheme. It is at this point that we effect the basic integration of the Mackey Machine and Orbit Method that activates the scheme. We also prove here that the little homogeneous spaces are symmetric. In section 6 we prove Theorem 1.1, and in section 7 we prove Theorems 1.2 and 1.3. We present in section 8 another suggestive example and two conjectures. Finally, section 9 is an appendix containing an argument supplied by the referee.

2. Elementary examples and observations. In this section we initiate our study by exhibiting a few of the simplest examples of orbital and quasi-orbital symmetric spaces. We also examine the effect of passing to a component stability group. Finally, we investigate the likelihood of the phenomena occurring for several standard types of groups.

Examples 2.1. (i) Abelian symmetric spaces. G = H ⋉ V, V a simply connected abelian normal Lie group (i.e. a vector subgroup), H arbitrary except V̂ /H is countably separated. Then by [11,Thm.2.1], we have

τ = ∫_{V̂ /H}^{⊕} π_{χ} dχ̇,

where dχ̇ is the pseudo-image measure class of Lebesgue measure dχ and π_{χ} = π_{1,χ} = Ind_{H_{χ}V}^{G} 1 × χ. If the measure dχ̇ is concentrated in a finite number of points, then G/H is quasi-orbital; and it is orbital if dχ̇ is concentrated in a point. The latter occurs of course if there is a Zariski-open orbit; and the former if there is a finite number of open orbits with conull union. We illustrate the simplest examples of both situations:

(ia) G = \{( \begin{matrix}a & b \\ 0 & 1 \end{matrix} ) : a ≠ 0, b ∈ ℝ\} , σ = conjugation by ( \begin{matrix}1 & 0 \\ 0 & −1 \end{matrix} ) ,

H = \{( \begin{matrix}a & 0 \\ 0 & 1 \end{matrix} ) : a ≠ 0\}. τ is irreducible.

(ib) Let G be as in (ia) and take the component group G^{0}. Then (G^{0})^{σ} = H^{0} and

τ = π_{+} ⊕ π_{−}, where π_{±} = Ind_{V}^{G}χ_{±1}, χ_{t} ( \begin{matrix}1 & b \\ 0 & 1 \end{matrix} ) = e^{itb}.

(ii) Strichartz symmetric spaces. Let G = R ⋉ V, where V is a normal vector subgroup and R is reductive. Suppose σ is an involution of G leaving both R and V invariant. Set S = R^{σ}, U = V^{σ}, H = G^{σ} = S ⋉ U. Then from [19,Thm.3.1], we obtain:

τ = ∫_{U^{⊥}/S}^{⊕} ∫_{R̂_{λ}(S_{λ})}^{⊕} n_{λ}(ν)π_{ν,λ} dμ_{λ}(ν)dλ̇;

where dλ̇ is the pseudo-image class of Lebesgue measure dλ (on U^{⊥}); (R̂_{λ}(S_{λ}), n_{λ}, μ_{λ}) is the Plancherel data in the “little homogeneous space” decomposition

τ_{λ} = Ind_{S_{λ}}^{R_{λ}}1 = ∫_{R̂_{λ}(S_{λ})}^{⊕} n_{λ}(ν)ν dμ_{λ}(ν);

and the irreducibles prescribed by the Mackey Machine are

π_{ν,λ} = Ind_{R_{λ}V}^{G} ν × λ, ν ∈ R̂_{λ}.

(R̂_{λ}(S_{λ}) refers to the ν’s weakly contained in τ_{λ}.) Thus G/H will be orbital exactly when U^{⊥}/S is a singleton (measure-theoretically) and R_{λ}/S_{λ} is orbital. We will prove (below in Lemma 5.1 — see also [19,Thm.5.3]) that R_{λ}/S_{λ} is symmetric — and so the outline of our inductive scheme begins to emerge. For example,

(iia) G = \{( \begin{matrix}a & b & x \\ c & d & y \\ 0 & 0 & 1 \end{matrix} ) : det g ≠ 0\} ≅ GL(2, ℝ) ⋉ ℝ^{2}. Let σ be conjugation by

the diagonal matrix with entries +1, −1, +1 respectively. Then H = S ⋉ U, with S =

( \begin{matrix}a & 0 & 0 \\ 0 & d & 0 \\ 0 & 0 & 1 \end{matrix} ) , U = ( \begin{matrix}1 & 0 & x \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix} ). U^{⊥}/S is a point (measure-theoretically) and the little

homogeneous space is exactly (ia).

(iib) If we replace GL(2, ℝ) in the last example by GL^{+}(2, ℝ), i.e. positive determinant, then U^{⊥}/S is still one point, but R_{λ}/S_{λ} is now quasi-orbital.

(iic) The reverse of (iib) can happen. For example, take R = ( \begin{matrix}a & 0 \\ 0 & d \end{matrix} ) , a ≠ 0, d > 0.

Then U^{⊥}/S becomes two points while R_{λ} = S_{λ}.

Finally, here is an example wherein the unipotent radical is really nilpotent, not abelian. (iii) Let 𝔤 be the 4-dimensional solvable Lie algebra with generators T, X, Y, Z satisfying

bracket relations

[X, Y] = Z [T, X] = X [T, Z] = Z.

Define σ by σ : T → T, X → X, Y → −Y, Z → −Z. Let 𝔥 = 𝔤^{σ} and let G, H be the corresponding simply connected groups. Using the Orbit Method [15], it is easy to check that G/H is quasi-orbital. We leave it to the reader to render the example orbital by enlarging G to a disconnected group with the same Lie algebra.

We conclude this preliminary section with two results suggested in part by the above (and other) examples.

Proposition 2.2. Let σ be a non-trivial involution of a Lie group G, H = G^{σ}. If G is connected exponential solvable, then τ is not orbital. If G is nilpotent or reductive, and non-compact, then τ cannot be quasi-orbital.

Proof. First suppose that G is simply connected exponential solvable (in which case H is also). In this case one knows the direct integral decomposition of τ is given by

τ = ∫_{𝔥^{⊥}/H}^{⊕} π_{ϕ} dϕ̇,

(see [10]). But it follows easily from [5,Ch.I] that the exponential action of H on the vector space 𝔥^{⊥} cannot result in a single conull orbit. Now suppose G is only connected, but not simply connected, and τ = Ind_{H}^{G}1 is irreducible. Consider the simply connected covering group G̃. Then G = G̃/D, where D is a discrete subgroup of Z̃ = Cent G̃. Any involution σ differentiates to 𝔤 and then lifts to G̃. If we denote by p : G̃ → G the canonical projection, then we have a commutative diagram

G̃ ⟶^{σ} G̃
p↓  ^{p}↓
G ⟶^{σ} G.

In particular σ preserves D. Write H̃ = G̃^{σ} and consider p^{−1}(H), which is easily seen to satisfy

p^{−1}(H) = {g̃ : g̃^{−1}σ(g̃) ∈ D} .

We use the fact that in the exponential solvable case one knows that G̃ = H̃ exp 𝔮 (see [4]). Therefore if g̃ = h̃ exp X ∈ p^{−1}(H), then g̃^{−1}σ(g̃) is in D and also equals exp −2X. Hence 2X ∈ 𝔷 ⇒ X ∈ 𝔷 ⇒ p^{−1}(H) ⊂ H̃Z̃. Suppose the inequality is strict. We have (Ind_{H}^{G}1)∘p = IndG̃ ; p^{−1}(H)1. But the latter is not irreducible, since Ind_{p^{−1}(H)}^{H̃Z̃}1 is not irreducible. This is a contradiction. On the other hand, if p^{−1}(H) = H̃Z̃, then since H and p^{−1}(H) have the same Lie algebra, this would force Z̃ ⊂ H̃, i.e. p^{−1}(H) = H̃. Then (Ind_{H}^{G}1) ∘ p = IndG̃ ; H̃1, and we are reduced back to the simply connected case.

If G is nilpotent, the H-orbits in 𝔥^{⊥} are closed, and therefore lower-dimensional. Thus if G is simply connected, τ cannot be quasi-orbital. Basically the same argument as above shows that this fact extends to the non-simply connected case — the only change would be that quasi-orbital forces p^{−1}(H) to be of finite index in H̃Z̃, therefore open, and therefore everything since the latter is connected.

Finally, if G is reductive the result is clear from well-known results on semisimple symmetric spaces (see [3]).

The second result is the following

Proposition 2.3. Let G be algebraic and suppose that H is a proper open subgroup of G^{σ}. Then Ind_{H}^{G}1 cannot be orbital.

Proof. Because of the algebraic assumption, the component group G^{σ}/(G^{σ})^{o} is finite. Then Ind_{H}^{G^{σ}}1 cannot be irreducible. It follows that Ind_{H}^{G}1 = Ind_{G^{σ}}^{G}Ind_{H}^{G^{σ}}1 cannot be irreducible.

Remarks. It would be interesting to know if the algebraic assumption could be dropped in Prop. 2.3. Without it, the component group could be really nasty, and a quasi-regular representation of a “nasty” discrete group could be irreducible. We also remark that, as Example 2.1(ib) shows, the proposition cannot be strengthened to quasi-orbital.

3. Two auxiliary results. In this section we prove two (unrelated) results. The first will be extremely useful in the sequel. Both may be useful in other contexts as well. The first result says that any involution of an algebraic group is made up of a Levi and unipotent part. By an algebraic group G we shall always mean in this paper that G is the set of real points G(ℝ) of a complex algebraic group G def/ℝ. If N is the unipotent radical of G, we call N = N(ℝ) the unipotent radical of G. By a Levi component R of G we mean R = R(ℝ) for any Levi component R of G that is def/ℝ. Now let G be algebraic, N its unipotent radical. Since N is characteristic any involution of G preserves N. But we also have

Theorem 3.1. Let σ be an involution of the algebraic group G. Then there is a Levi component of G that is preserved by σ.

Proof. We first prove the result when N is abelian. Select a Levi component R. Then σ(R) is another Levi component. But any two Levi components are conjugate by an element of N. So σ(R) = nRn^{−1} for some n ∈ N.

Lemma 3.2. Suppose σ(R) = nRn^{−1} and σ(n) = n. Then nRn^{−1} = R, i.e. R is σinvariant.

Proof. We have R^{2} = σ^{2}(R) = σ(σ(R)) = σ(nRn^{−1}) = σ(n)σ(R)σ(n)^{−1} = n^{2}Rn^{−2}. That is, m = n^{2} normalizes R. So for any r ∈ R, we have

r^{−1}mrm^{−1} ∈ R ∩ N ⇒ r^{−1}mrm^{−1} = 1.

That is, m ∈ Z_{G}(R). Let m = exp X, X ∈ 𝔫. It must be that X ∈ ℨ_{𝔤}(R). But the set of vectors in 𝔫 which are fixed by R is a vector space. Hence \frac{1}{2}X ∈ ℨ_{𝔤}(R) ⇒ n ∈ Z_{G}(R). □

Now continuing with the proof of Theorem 3.1, we have σ(R) = nRn^{−1}. Write

n = pq, p, q ∈ N, σ(p) = p, σ(q) = q^{−1}.

Choose ℓ ∈ exp(𝔮 ∩ 𝔫) satisfying ℓ^{2} = q. Then we can compute

σ(ℓRℓ^{−1}) = σ(ℓ)σ(R)σ(ℓ)^{−1}
= ℓ^{−1}nRn^{−1}ℓ
= nℓ^{−1}Rℓn^{−1} (N abelian)
= pℓRℓ^{−1}p^{−1}.

Applying Lemma 3.2 to ℓRℓ^{−1} and p, we obtain σ(ℓRℓ^{−1}) = ℓRℓ^{−1}.

We turn at last to the general case of unipotent N. The proof is by induction on dim N. If dim N = 1, then N is abelian and we are done. Otherwise, let Z = Cent N. Z is both R and σ-invariant. Consider G_{1} = G/Z. The unipotent radical of G_{1} is N/Z which has lower dimension than that of N. Moreover, σ passes to G_{1}, call the resulting involution σ_{1}. If p : G → G_{1} is the canonical projection, we have a commutative diagram

G ⟶^{σ} G
p↓  ^{p}↓
G_{1} ⟶^{σ_{1}} G_{1}.

Let R be any Levi component of G. Then R_{1} = p(R) is a Levi component of G_{1} and p : R → R_{1} is an isomorphism. By induction, we can choose R′_{1} conjugate to R_{1}, a Levi subgroup of G_{1}, which is σ_{1}-invariant. Suppose R′_{1} = n_{1}R_{1}n_{1}^{−1}, n_{1} ∈ N/Z. Select any n ∈ N satisfying p(n) = n_{1}. Then

p(nRn^{−1}) = p(n)p(R)p(n)^{−1} = n_{1}R_{1}n_{1}^{−1} = R′_{1}.

Therefore

p^{−1}(R′_{1}) = nRn^{−1}Z,

and since R′_{1} is σ_{1}-invariant, it follows easily that the group G_{2} = nRn^{−1}Z is σ-invariant. Then we apply the case of abelian unipotent radical to the group G_{2}. The resulting σ-invariant Levi subgroup of G_{2} will be a σ-invariant Levi subgroup of G.

The second result is a strong form of Frobenius reciprocity for orbital symmetric spaces. In order to present it we need further notation which will also be important in the rest of the paper. For H ⊂ G we choose right Haar measures dh, dg on H and G, and we also choose a smooth function q satisfying

q(hg) = \frac{Δ_{H}(h)}{Δ_{G}(h)}q(g), h ∈ H, g ∈ G,

Δ denoting Haar measures. Then a quasi-invariant measure dġ on H\G is determined by

∫_{G} f(g)q(g) dg = ∫_{H\G} ∫_{H} f(hg) dhdġ.

If χ is a character of H, the induced representation τ_{χ} = Ind_{H}^{G}χ acts in the space

L^{2}(G; H; χ) = {f : G → ℂ, f(hg) = χ(h)f(g), ∫_{H\G} |f|^{2} dġ < ∞}

by the action

τ_{χ}(g)f(x) = f(xg)[q(xg)/q(x)]^{1/2}.

If q is a character on all of G, then dġ is relatively invariant with modulus q. More generally, if α is an automorphism of G preserving H, then we shall write q_{H,G} for the modulus of the action of α on H\G,

q_{H,G}(α) ∫_{H\G} f(α(g)) dġ = ∫_{H\G} f(g) dġ.

Now we take our motivation from the basic result of Penney [20]. If τ = Ind_{H}^{G}1 is type I and has a direct integral decomposition

τ = ∫ ⊕ ; Ĝ n(π)π dμ_{τ} (π)

with respect to a Plancherel measure class μ_{τ}, then the multiplicity function n(π) satisfies

(3.1)  n(π) ≤ dim (H_{π}^{−∞})^{H,q^{−1/2}} , μ_{τ} − a.e.

Here H_{π} denotes a space on which π acts: H_{π}^{∞} is the dense subspace of C^{∞} vectors;

H_{π}^{−∞} is the anti-dual space of distributions; and (H_{π}^{−∞})^{H,q^{−1/2}} denotes the subspace of distributions which transform under H according to q^{−1/2}, where q is the quotient of the modular functions of H and G as above. The strong (distribution-theoretic) form of Frobenius reciprocity is equality in formula (3.1). Now if τ is irreducible, then the ideal statement of reciprocity would be that the right side of (3.1) is 1 precisely for the class τ and 0 otherwise. Indeed we have

Theorem 3.3. Let G/H be an orbital symmetric space. Let π ∈ Ĝ. Then

dim (H_{π}^{−∞})^{H,q^{−1/2}} = \{ \begin{matrix}1 & π ≅ τ \\ 0 & otherwise. \end{matrix}

Proof. We first observe that the canonical cyclic distribution

α : f → f̄(1)

is a member of (H_{τ}^{−∞})^{H,q^{−1/2}} [16,Prop.2.2]. We show that any other distribution is a

multiple of it. Let β ∈ (H_{τ}^{−∞})^{H,q^{−1/2}}. We associate to β a distribution β̃ on G defined by

⟨β̃, ϕ⟩ = ⟨β̄, ϕ̃⟩
ϕ̃(g) = ∫_{H} ϕ(hg)q^{−1/2}(hg) dh, ϕ ∈ D(G).

Here, D(G) denotes the space of smooth, compactly supported (i.e. test) functions on G. Of course ϕ̃ is a C^{∞} vector for τ, so β̃ is well-defined. We observe that β̃ transforms under the action of H according to

(3.2)  ⟨β̃, ϕ_{h}⟩ = Δ_{H}^{1/2}(h)Δ_{G}^{1/2}(h)⟨β̃, ϕ⟩
(3.3)  ⟨β̃, ϕ^{h}⟩ = q^{1/2}(h)⟨β̃, ϕ⟩

where as usual ϕ_{h}(x) = ϕ(h^{−1}x), ϕ^{h}(x) = ϕ(xh). (We have used the easily verified equations ϕ̃_{h} = Δ_{H}^{1/2}(h)Δ_{G}^{1/2}(h)ϕ̃, (ϕ^{g})^{∼} = τ(g)ϕ̃.)

Now, given any intertwining operator A for τ, we can define a distribution β_{A} on G by

⟨β_{A}, ϕ⟩ = (Aϕ̃)(1), ϕ ∈ D(G).

But β_{A} satisfies the same equivariance conditions (3.2), (3.3). Indeed (3.2) is obvious, and (3.3) follows from the computaion

⟨β_{A}, ϕ^{h}⟩ = A(ϕ^{h})^{∼}(1) = Aτ(h)ϕ̃(1)
= τ(h)Aϕ̃(1) = Aϕ̃(h)q^{1/2}(h)
= Aϕ̃(1)q^{1/2}(h) = q^{1/2}(h)⟨β_{A}, ϕ⟩.

Now the point of [21, Theorem 2] is that the set of distributions on G satisfying (3.2) and (3.3) is in 1-1 correspondence with the set of intertwining operators. Then we invoke the irreducibility of τ to know that the only intertwining operators are scalars. Therefore, every distribution satisfying (3.2) and (3.3) is a scalar multiple of β_{I} : ϕ → ϕ̃(1) = ∫_{H} q^{1/2}(h)ϕ(h) dh. In particular, for β ∈ (H_{τ}^{−∞})^{H,q^{−1/2}}, we must have β̃ = cβ_{I}, i.e.⟨β̃, ϕ⟩ = ⟨β̄, ϕ̃⟩ = cϕ̃(1) = c⟨ᾱ, ϕ⟩.

Now let π be an irreducible unitary representation of G, inequivalent to τ. Let β ∈ (H_{π}^{−∞})^{H,q^{−1/2}}. We shall show that β = 0. First of all, since π is irreducible, β must be cyclic. Then we can use the canonical realization (π̃, β̃) of (π, β) (see [20,p. 183]). In that realization the Hilbert space H̃_{π} of π is a space of distributions on D(G). Moreover, according to [20,Prop.II.3], the C^{∞} vectors in that realization will consist of a collection of smooth locally integrable functions, and β̃ is evaluation at the identity. But since β̃ is H-equivariant, the functions in the Hilbert space H̃_{π} must be H-invariant. We also know (from [20,Theorem A]) that the action of π on H̃_{π}^{∞} is irreducible. All of this together implies that, since π ≅ τ̸, the space H̃_{π} is trivial, and therefore β = 0.

In fact we can also prove the converse.

Theorem 3.4. Suppose H ⊂ G (not necessarily symmetric) and τ = Ind_{H}^{G}1 is type I. Suppose there exists an irreducible unitary representation π_{1} ∈ Ĝ such that

dim (H_{π}^{−∞})^{H,q^{−1/2}} = \{ \begin{matrix}0 & π ≅ π̸_{1} \\ 1 & π ≅ π_{1}. \end{matrix}

Then τ is irreducible and equivalent to π_{1}.

Proof. If τ is not irreducible, the type I condition means that τ = ∫_{X}^{⊕} π dμ(π), a direct integral of irreducibles. By [20] we also have

α_{τ} = ∫_{X}^{⊕} α_{π} dμ(π).

Moreover, the H-equivariance condition is preserved by the direct integral, so α_{π} ∈ (H_{π}^{−∞})^{H,q^{−1/2}} , μ − a.a. π. But by hypothesis, no distribution space other than the one corresponding to π_{1}, contains any non-zero elements. Therefore it must be that π ≅ π_{1} for μ − a.a. π ∈ X. That is, τ is a multiple of π_{1}. But if τ = nπ_{1}, n ≥ 1, then again by (3.1), n ≤ dim (H_{π}^{−∞})^{H,q^{−1/2}} = 1. □

4. The orbital interpretation. In order to justify the name, we would like to bring the family of orbital symmetric spaces under the rubric of the Orbit Method. In particular, we would like to characterize — in orbital terms — these spaces. In this section we formulate precise orbital criteria for G/H to be orbital or quasi-orbital. In the next two sections we set up the inductive scheme required for the proof of the criteria.

We take our cue from prior work on orbital formulations of harmonic analysis on homogeneous spaces([12] [15]). These papers suggest the orbital integral formulae

(4.1)  τ = Ind_{H}^{G}1 = ∫_{𝔥^{⊥}/H}^{⊕} π_{ϕ} dϕ̇ = ∫_{G·𝔥^{⊥}/G}^{⊕} n_{ϕ}π_{ϕ} dϕ̈,

where 𝔥^{⊥} = {ϕ ∈ 𝔤^{∗} : ϕ(𝔥) = 0}, n_{ϕ} = #(G · ϕ ∩ 𝔥^{⊥})/H, and dϕ̇, dϕ̈ are pseudo-images of Lebesgue measure dϕ on 𝔥^{⊥}. Thus for orbital symmetric spaces one would expect that 𝔥^{⊥}/H should be one point measure-theoretically; and finitely many points for quasiorbital spaces. Taking into account that the dual of the component group G_{ϕ}/(G_{ϕ})^{0} also contributes parameters (in the Orbit Method [1] [5]), we are led to the following Theorem 4.1. Let G be algebraic, σ an involution of G, H = G^{σ}. Then G/H is

(i) orbital iff there exists a Zariski-open H-orbit on 𝔥^{⊥} and for any point ϕ in that orbit, G_{ϕ} ⊂ H;

(ii) quasi-orbital iff there exists finitely many open H-orbits in 𝔥^{⊥} whose union is conull.

We prove this result by the inductive technique established in the next section. As explained in the introduction, the technique is just as important as the result, and we shall use it to get another important theorem later in the paper. But first we make several elementary observations. In fact, the existence of an open H-orbit on 𝔥^{⊥} being related to irreducibility of τ is quite plausible — even aside from the motivation supplied by the orbital integral formulae (4.1). For we have

Proposition 4.2. Let ϕ be in an open H-orbit on 𝔥^{⊥}. Then 𝔥 is maximal totally isotropic with respect to the bilinear form B_{ϕ}(X, Y) = ϕ[X, Y], i.e. 𝔥 is a real polarization for ϕ.

Proof. Since H·ϕ is open in 𝔥^{⊥} we must have 𝔥·ϕ = 𝔥^{⊥}. Then dim 𝔥/𝔥_{ϕ} = dim 𝔤/𝔥 = dim 𝔮, where 𝔤 = 𝔥 + 𝔮 is the ±1 eigenspace decomposition of 𝔤 with respect to σ.

Lemma 4.3. For any ϕ ∈ 𝔥^{⊥}, we have 𝔤_{ϕ} = 𝔥_{ϕ} + 𝔮_{ϕ}.

Proof. Let X ∈ 𝔤_{ϕ}, X · ϕ = 0. Write X = Y + Z ∈ 𝔥 + 𝔮. Then (Y + Z) · ϕ = 0. Apply σ to obtain (Y − Z) · −ϕ = 0. Combining these equations we get Y · ϕ = Z · ϕ = 0.

We observe that 𝔥 is clearly totally isotropic for B_{ϕ} (since ϕ ∈ 𝔥^{⊥}). Thus we can complete the proof of Proposition 4.2 by showing that 𝔥_{ϕ} = 𝔤_{ϕ} when H · ϕ is open. If that is not the case, then 𝔥 + 𝔮_{ϕ} is a subordinate subspace for B_{ϕ} containing 𝔤_{ϕ}. Therefore

dim(𝔥 + 𝔮_{ϕ})/𝔤_{ϕ} ≤ dim 𝔤/(𝔥 + 𝔮_{ϕ}).

Applying Lemma 4.3, we obtain

dim 𝔥/𝔥_{ϕ} ≤ dim 𝔮/𝔮_{ϕ}.

But we already saw that dim 𝔥/𝔥_{ϕ} = dim 𝔮. Hence 𝔮_{ϕ} = 0.

The theory of real polarizations and induced representations suggests that the representation π_{ϕ} = Ind_{H}^{G}χ_{ϕ}, χ_{ϕ}(exp X) = e^{iϕ(X)}, which is precisely τ since ϕ(𝔥) = 0, should be irreducible — at least if H includes the stability group. Taken together, these remarks provide further corroboration for the plausibility of Theorem 4.1. Also, included in the above proof is a result which will be needed in the proof of Theorem 6.1.

Corollary 4.4. If ϕ ∈ 𝔥^{⊥} lies in an open H-orbit, then 𝔤_{ϕ} = 𝔥_{ϕ}.

I close this section with two remarks about differential operators. The first is that if G/H is orbital, then D(G/H) — the algebra of G-invariant differential operators on G/H — must be trivial. But, unlike in the Riemannian or nilpotent situations, G/H may not be diffeomorphic to exp 𝔮 and in fact the algebra of H-invariants D(𝔮)^{H} may be non-trivial. When H is not reductive, the existence of an open H-orbit on 𝔥^{⊥} does not imply the same on 𝔮. The reader can verify these claims by means of Example 2.1(iia). Now the fact that the algebra of unbounded operators D(G/H) is trivial when τ is irreducible, although strongly suggested by the triviality of the commutatant (of bounded operators) of τ, is not obvious. The referee has supplied a proof which is found in section 9.

5. The inductive technique. The immediate goal is a proof of Theorem 4.1. Suppose G = R ⋉ N is algebraic, σ an involution of G. By Theorem 3.1, it is no loss of generality to assume R is σ- invariant. Then H = G^{σ} = S ⋉ M, S = R^{σ}, M = N^{σ}. Now the strategy is to apply the Mackey Machine to the analysis of τ. We shall see that it is possible to develop an inductive procedure or scheme which relates issues for the symmetric space G/H to corresponding matters on the “little homogeneous spaces.” We shall construct the scheme by stirring the Mackey Machine output into the analysis of τ in situations of successively greater generality. At each stage we add some new ingredients into the mixture (mainly from [1] [6] [7] [13] and [19]) that allow the structure to take shape. The stages are marked by the weaker and weaker assumptions of: (a) abelian unipotent radical, (b) no Mackey obstructions, (c) no central twist, and (d) full generality. Stages (a) and (b) can be thought of as “first approximations.” Stage (c) contains the guts of the computation, but falls slightly short of the goal since the little homogeneous spaces that appear do not have exactly the same structure as the original. This is remedied in stage (d), where very little new computation is required, but a reformulation of the original problem is critical to rendering the situation completely hereditary.

5a. Abelian unipotent radical. This computation appears in [19,§3] (see also Example 2.1(ii)). To emphasize that N is abelian, we temporarily write V in place of N and U = V^{σ}. The computation proceeds as follows.

τ = Ind_{SU}^{RV} 1 = Ind_{SV}^{RV} Ind_{SU}^{SV} 1
= Ind_{SV}^{RV} ∫_{U^{⊥}/S}^{⊕} Ind_{S_{λ}V}^{SV} 1 × λ dλ̇ (by [11])

= ∫_{U^{⊥}/S}^{⊕} Ind_{S_{λ}V}^{RV} 1 × λ dλ̇

= ∫_{U^{⊥}/S}^{⊕} Ind_{R_{λ}V}^{RV} Ind_{S_{λ}V}^{R_{λ}V} 1 × λ dλ̇

= ∫_{U^{⊥}/S}^{⊕} Ind_{R_{λ}V}^{RV} ∫_{R̂_{λ}(S_{λ})}^{⊕} n_{λ}(ν)ν × λ dλ̇

= ∫_{U^{⊥}/S}^{⊕} ∫_{R̂_{λ}(S_{λ})}^{⊕} n_{λ}(ν)π_{λ,ν} dμ_{λ}(ν)dλ̇ (5a.1)

= ∫_{R·U^{⊥}/R}^{⊕} ∫_{R̂_{λ}(S_{λ})}^{⊕} n_{λ}n_{λ}(ν)π_{λ,ν} dμ_{λ}(ν)dλ̈ (5a.2)

Here: dλ̇, dλ̈ are the measure classes of the pseudo-images of Lebesgue measure dλ; (R̂_{λ}(S_{λ}), n_{λ}, dμ_{λ}) are the Plancherel constituents of the “little” quasi-regular representation

(5a.3)  τ_{λ} = Ind_{S_{λ}}^{R_{λ}}1 = ∫_{R̂_{λ}(S_{λ})}^{⊕} n_{λ}(ν)ν dμ_{λ}(ν);

and n_{λ} = #(R · λ ∩ U^{⊥})/S. (See [19,§§3-5] for a discussion of how the symmetry of G/H simplifies the computation.) The spaces R_{λ}/S_{λ} are symmetric (see [19,Thm.5.3]), and they have lower dimension than that of G/H. So as we said in §2, the contours of the inductive technique begin to be evident. That is, if we wish to derive some property of τ, we can assume by induction that all the τ_{λ}’s have it (usually manifested in (5a.3)), and then try to control the outer integral in (5a.1) or (5a.2).

5b. No Mackey obstructions. Now we want to extend to non-abelian unipotent radical. To see how the computation of part (a) might generalize, we temporarily make two dramatic, and ultimately untenable, assumptions. The first of them is that: generically on N̂(M), any representation γ extends to an ordinary representation γ̃_{R} × γ of R_{γ}N. In that case we can do the following. We know by the nilpotent theory that

τ_{N} = Ind_{M}^{N}1 = ∫_{N̂(M)}^{⊕} γ dμ_{M}(γ),

a multiplicity-free decomposition ([6] [12]). In fact: N̂(M) ↔ 𝔪^{⊥}/M; μ_{M} ↔ dθ̇; the multiplicity-free condition arises because N · θ ∩ 𝔪^{⊥} = M · θ for any θ ∈ 𝔪^{⊥}; and

τ_{N} = ∫_{𝔪^{⊥}/M}^{⊕} γ_{θ} dθ̇.

Now in attempting to mimic the computation in (a), we have

τ = Ind_{H}^{G}1 = Ind_{SM}^{RN}1 = Ind_{SN}^{RN}Ind_{SM}^{SN}1.

The inside representation Ind_{SM}^{SN}1 is exactly the subject matter of [13,Thm.5.1]. The result is that for μ_{M} − a.a. γ, there is a canonical unitary intertwining action γ̃_{S} of S_{γ} on the space of γ so that

Ind_{SM}^{SN}1 = ∫_{N̂(M)/S}^{⊕} Ind_{S_{γ}N}^{SN} γ̃_{S} × γ dμ̇_{M}(γ).

The second of our excessive assumptions is that γ̃_{R}|_{S_{γ}} = γ̃_{S} (so that in particular we can drop the subscript and write unambiguously γ̃). But let us be quite clear: γ̃ exists naturally on S_{γ}; it is the extension to R_{γ} that is entirely too much to expect. Nevertheless, we continue with the computation

τ = Ind_{H}^{G}1 = Ind_{SN}^{RN} ∫_{N̂(M)/S}^{⊕} Ind_{S_{γ}N}^{SN} γ̃_{S} × γ dμ̇_{M}(γ)

= ∫_{N̂(M)/S}^{⊕} Ind_{S_{γ}N}^{RN} γ̃ × γ dμ̇_{M}(γ)

= ∫_{N̂(M)/S}^{⊕} Ind_{R_{γ}N}^{RN}Ind_{S_{γ}N}^{R_{γ}N} γ̃ × γ dμ̇_{M}(γ).

Now it is not too difficult to show: suppose that the quasi-regular representations τ_{γ} = Ind_{S_{γ}}^{R_{γ}} 1 are type I and have direct integral decompositions

(5b.1)  τ_{γ} = Ind_{S_{γ}}^{R_{γ}} 1 = ∫_{R̂_{γ}(S_{γ})}^{⊕} n_{γ}(ν)ν dμ_{γ}(ν);

then

(5b.2)  Ind_{S_{γ}N}^{R_{γ}N} γ̃ × γ ≅ ∫_{R̂_{γ}(S_{γ})}^{⊕} n_{γ}(ν)ν ⊗ γ̃ × γ dμ_{γ}(ν).

Since these computations are all made presuming unrealistic hypotheses, and since I shall not actually need (5b.2) in the sequel, I omit the proof. But I do include the proof of Lemma 5.1. R_{γ}/S_{γ} is symmetric.

Proof. Any γ ∈ N̂(M) can be associated to some θ ∈ 𝔪^{⊥} (we write γ = γ_{θ}). Then σθ = −θ and σ preserves G_{θ}. Indeed if g ∈ G_{θ}, then θ(g · X) = θ(X), ∀X ∈ 𝔫. Replacing θ by −σθ we get −θ(σ(g · X)) = −θ(σ(X)). That is, θ(σ(g) · σ(X)) = θ(σ(X)), which says σ(g) ∈ G_{θ}. But G_{γ} = G_{θ}N, so σ preserves G_{γ}. Then R_{γ} = G_{γ} ∩ R, and the σ-invariance of R implies that R_{γ} is also invariant. Clearly R_{γ}^{σ} = R_{γ} ∩ R^{σ} = R_{γ} ∩ S = S_{γ}. We note for future reference that we have also proven that the spaces G_{θ}/H_{θ}, θ ∈ 𝔪^{⊥}, are symmetric.

Now we can conclude the computation of the direct integral decomposition:

(5b.3)  τ = Ind_{H}^{G}1 = ∫_{N̂(M)/S}^{⊕} ∫_{R̂_{γ}(S_{γ})}^{⊕} n_{γ}(ν)π_{γ,ν} dμ_{γ}(ν)dμ̇_{M}(γ),

where π_{γ,ν} are the Mackey parameters for the irreducible π_{γ,ν} = Ind_{R_{γ}N}^{G}ν ⊗ γ̃ × γ. Finally, the proof of Lemma 5.1 shows that if r · γ_{1} = γ_{2}, then H_{γ_{2}} = σ(r)H_{γ_{1}}σ(r)^{−1}. (See also [19,Thm.5.3(ii)].) Therefore, the final expression is

(5b.4)  τ = ∫_{R·N̂(M)/R}^{⊕} ∫_{R̂_{γ}(S_{γ})}^{⊕} n_{γ}n_{γ}(ν)π_{γ,ν} dμ_{γ}(ν)dμ̈_{M}(γ),

where n_{γ} = #(R · γ ∩ N̂(M))/S.

Hence, as in part (a), we have the makings of an inductive technique — i.e. a passage from the symmetric space G/H to the lower dimensional little symmetric spaces R_{γ}/S_{γ}. In fact a formal proof of Theorem 4.1 could be built around (5b.3) and (5b.4). But unfortunately we must acknowledge the unlikelihood of the assumptions we made on γ̃ being satisfied. Basically they only hold when the Mackey obstructions to γ̃_{R} existing as an ordinary representation of R_{γ} vanish. Typically the obstructions don’t vanish — they have two components: an obstruction to extending the character χ_{θ} (see below) from N_{θ} to G_{θ}; and the possible failure of the existence of a G_{θ}-invariant polarization in Sp(𝔫/𝔫_{θ}) — which causes a 2-fold obstruction to appear. The means of dealing with these obstructions, as well as their integration into the Orbit Method framework, were accomplished simultaneously by Duflo in [7]. It is our purpose to further integrate that mechanism with the symmetric space structure.

5c. No central twist. Henceforth, we set K = SN. Let θ ∈ 𝔫^{∗} and γ = γ_{θ} ∈ N̂ the corresponding representation. γ_{θ} is constructed as follows. Starting from the character χ_{θ} of N_{θ} defined by χ_{θ}(exp X) = e^{iθ(X)}, X ∈ 𝔫_{θ}, it is induced (ordinarily or holomorphically) via a (real or complex) polarization for the bilinear form B_{θ}(X, Y) = θ[X, Y] (see [10]). Then G_{γ} = R_{γ}N = G_{θ}N, and K_{γ} = S_{γ}N = K_{θ}N. Of course G_{θ} and K_{θ} may not split. There is a canonical 2-fold covering of these groups defined as follows. The group G_{θ} preserves the symplectic form B_{θ} on 𝔫/𝔫_{θ}, so there is a natural map

G_{θ} → Sp(𝔫/𝔫_{θ}).

Let Mp(𝔫/𝔫_{θ}) be the metaplectic cover of Sp(𝔫/𝔫_{θ}), with p the covering map. Then there is a commutative diagram

G̃_{θ} ⟶ Mp(𝔫/𝔫_{θ})
p↓  ^{p}↓
G_{θ} ⟶ Sp(𝔫/𝔫_{θ}),

where we can take G̃_{θ} = {(g_{θ}, m) ∈ G_{θ} × Mp(𝔫/𝔫_{θ}): the components have the same image in Sp(𝔫/𝔫_{θ})}. Alternatively, we have G̃_{θ} = {(g_{θ}, s) ∈ G_{θ} × ℂ : s^{2} = δ(g_{θ}) := det_{𝔫_{c}/𝔟}(g_{θ})/| det_{𝔫_{c}/𝔟}(g_{θ})|}, where 𝔟 is any polarization for θ, stable under g_{θ}. We set δ^{1/2}(g_{θ}, s) = s. The notation is valid since δ^{1/2}(g_{θ}, s)^{2} = δ(g_{θ}). It is evident now that G̃_{θ} is a 2-fold cover of G_{θ}. We set ϵ = the non-trivial element in ker p, δ^{1/2}(ϵ) = −1. δ^{1/2} may or may not be a character of G̃_{θ}. It is if there exists a single polarization invariant under all of G_{θ}.

The group G̃_{θ} acts on 𝔫 (by factoring through p), and a fundamental result of Duflo [7,Thm.6.1] asserts that there is a canonical unitary representation γ̃_{θ} of G̃_{θ} on the space of γ so that

(5c.1)  γ̃_{θ}(g_{θ}, s)γ(n)γ̃_{θ}(g_{θ}, s)^{−1} = γ(g_{θ}ng_{θ}^{−1})

(5c.2)  γ̃_{θ}(n_{θ}, s) = χ̃_{θ}(n_{θ}, s)^{−1}γ(n_{θ}),

where χ̃_{θ}(n_{θ}, s) is the unique character of Ñ_{θ} satisfying χ̃_{θ}|_{N_{θ}} = χ_{θ}, χ̃_{θ}(ϵ) = −1. γ̃_{θ} is defined as follows. For any g_{θ} ∈ G̃_{θ} we may choose (as above) a positive complex polarization 𝔠 for θ which is invariant by g̃_{θ}. We realize γ_{θ} by holomorphic induction via 𝔠, i.e. γ_{θ} = 𝔠 − Ind_{N_{θ}}^{N}χ_{θ}, which acts in the subspace of functions (in the space of Ind_{N_{θ}}^{N}χ_{θ}) which satisfy X ∗ f = iθ(X)f, X ∈ 𝔠. Then γ̃_{θ}(g̃_{θ}) is defined by

γ̃_{θ}(g̃_{θ})f(n) = δ^{−1/2}(g̃_{θ})q_{D,N}^{−1/2}(g̃_{θ})f(g_{θ}^{−1}ng_{θ}),

where 𝔡 = 𝔠 ∩ 𝔫. Let us not forget that 𝔠 may vary with g_{θ}. (But the fundamental result of [1,Thm.II.3.1] or [7,Thm.3.1] is that if we fix a single realization of γ and conjugate all the operators γ̃_{θ}(g̃_{θ}) over to that realization, then γ̃_{θ} is uniquely determined.)

Note that G_{θ} usually does not split in G̃_{θ}, but that N_{θ} always splits in Ñ_{θ} (since δ(n_{θ}) ≡ 1). Then the representations of G lying over γ = γ_{θ} are obtained by

π_{γ,ν} = Ind_{G_{θ}N}^{G}ν ⊗ γ̃_{θ} × γ_{θ},

where ν is any irreducible representation of G̃_{θ} satisfying ν(n_{θ}, s) = χ̃(n_{θ}, s)Id. Thus the little group spectrum is parameterized by the spectrum of IndG̃_{θ} ; Ñ_{θ} χ̃_{θ} — which we write as

G̃_{θ}(Ñ_{θ}, χ̃_{θ})^{∧}.

Now we bring in the symmetric space structure. Indeed, we do not need the entire set G̃_{θ}(Ñ_{θ}, χ̃_{θ})^{∧}. We only need the part that contributes to the quasi-regular representation τ. To see exactly what that is we need another

Lemma 5.2. Let θ ∈ 𝔪^{⊥}. Then (1) K_{θ} = H_{θ}N_{θ}

(2) K̃_{θ} = H̃_{θ}Ñ_{θ} = H̃_{θ}N_{θ}.

Proof. Since θ ∈ 𝔪^{⊥}, it satisfies σθ = −θ and N · θ ∩ 𝔪^{⊥} = M · θ.

(i) For k ∈ K_{θ}, we have k = sn ∈ SN so that n · θ = s^{−1} · θ ∈ N · θ ∩ 𝔪^{⊥}. Therefore n · θ = m · θ for some m ∈ M. But then k = sn = smm^{−1}n ∈ H_{θ}N_{θ}.

(ii) The first equation is clear from part (i). The second follows from the fact that N_{θ} splits in Ñ_{θ}. It is also evident that H̃_{θ} ∩ Ñ_{θ} = M̃_{θ} and that H_{θ} ∩ N_{θ} = M_{θ}.

We already observed in the proof of Lemma 5.1 that σ preserves G_{θ}. It is also clear that σ extends canonically to G̃_{θ} — namely, σ(g_{θ}, s) = (σg_{θ}, s). Obviously G̃_{θ}^{σ} = H̃_{θ} and the symmetric space G̃_{θ}/H̃_{θ} is the same as G_{θ}/H_{θ}.

Next we observe that χ_{θ} has a canonical extension to K_{θ} by setting it equal to 1 on H_{θ}. That is a consistent thing to do since on H_{θ} ∩ N_{θ} = M_{θ}, the character χ_{θ} is identically 1. Also, χ̃_{θ} has a canonical extension to K̃_{θ}. That is because, by [13], the representation γ has an extension to S_{γ}. By the Duflo theory, the only way that can happen is if δ^{1/2} is a character of K̃_{θ}. That does not assure the existence of a single K_{θ}-invariant polarization — just that δ^{1/2} is a character. Then we get χ̃_{θ} on K̃_{θ} by defining

χ̃_{θ}(h̃_{θ}ñ_{θ}) = δ^{1/2}(h̃_{θ})χ̃_{θ}(ñ_{θ}).

Lemma 5.3. Ind_{SM}^{SN}1 = ∫_{N̂(M)/S}^{⊕} Ind_{S_{γ}N}^{SN} χ̃_{θ}γ̃_{θ} × γ_{θ} dμ̇_{M}(γ_{θ}).

Proof. Note that χ̃_{θ}γ̃_{θ} is an ordinary representation of K̃_{θ} that passes to K_{θ}. So since S_{γ}N = K_{θ}N, and since χ̃_{θ}γ̃_{θ}|_{N_{θ}} = γ|_{N_{θ}}, the representation χ̃_{θ}γ̃_{θ} × γ_{θ} is a well-defined representation of K_{θ}N.

This lemma is proven in [13] in the case that SN is completely solvable — which translates here to S being split abelian. For the more general S we have here, the argument is somewhat lengthy. If we examine [12,§7] and [13,§5], we see from the general arguments there that the following is true:

(5c.3)  τ = Ind_{SM}^{SN}1 = ∫_{N̂(M)/S}^{⊕} Ind_{S_{γ}N}^{SN} γ̃ × γ dμ̇_{M}(γ)

where γ̃ is the representation (referred to in §5b as γ̃_{S}) of S_{γ} on the space H_{γ} of γ determined as follows. We have: (i) a multiplicity-free direct integral decomposition

(5c.4)  τ_{N} = Ind_{M}^{N}1 = ∫_{N̂(M)}^{⊕} γ dμ_{M}(γ);

an action of S on τ_{N} determined by

(5c.5)  ((τ̃_{N}(s)f) (n) = q_{N,M}^{−1/2}(s)f(s^{−1}ns), f ∈ L^{2}(N; M);

and therefore a uniquely determined family of operators

τ̃_{N,γ}(s) : H_{s·γ} → H_{γ}

satisfying

γ(n)τ̃_{N,γ}(s) = τ̃_{N,γ}(s)(γ · s)(s^{−1}ns), s ∈ S, n ∈ N.

Therefore γ̃ := τ̃_{N,γ}|_{S_{γ}} is a representation of S_{γ}; the representation γ̃ × γ is the canonical extension of γ to S_{γ}N specified by [13,Thm.5.1]; the representation π_{γ} = Ind_{S_{γ}N}^{SN} γ̃ × γ is irreducible; and we have a multiplicity-free decomposition (5c.3). All that we have to do is prove that

(5c.6)  τ̃_{N,γ} = χ̃_{θ}γ̃_{θ}.

We recall that S_{γ}N = K_{θ}N and so it will be enough to prove (5c.6) on K_{θ}. Furthermore, we know from Lemma 5.2 that K_{θ} = H_{θ}N_{θ}, so it will be sufficient to examine γ̃ on H̃_{θ}. Looking closely at the definition of χ̃_{θ} on H̃_{θ}, as well as the definition of γ̃_{θ} in [7], we see that the proof reduces to showing the following: Let

Φ : f → {f_{θ}}_{θ∈𝔪^{⊥}/M}

be the intertwining operator for the direct integral decomposition of τ_{N} (5c.4). The operator Φ is described explicitly in [16] — which description we shall use momentarily. For any θ ∈ 𝔪^{⊥}, let S be a group of automorphisms of N that fix θ. For s ∈ S define τ̃_{N}(s) according to (5c.5), i.e.

(τ̃_{N}(s)f) (n) = q_{N,M}^{−1/2}(s)f(s^{−1}ns).

Then the infinitesimal action of τ̃_{N} at the place θ, call it τ̃_{θ}(s), is precisely Φ_{θ}(τ̃(s)f). We must prove that

(5c.7)  τ̃_{θ}(s) = δ^{1/2}(s)γ̃_{θ}(s).

We do it by induction on dim N. Low dimensional cases (say less than three) correspond to abelian N and the argument is trivial. The induction argument itself is patterned after [7,pp. 102,3]. But before starting it, we examine the case of an invariant real polarization 𝔟. So suppose θ ∈ 𝔪^{⊥} is left invariant by a group of automorphisms S (of N), and that there also exists a real polarization 𝔟 for θ preserved by S.

In that case the representation γ_{θ} is realized by γ_{θ} = Ind_{B}^{N}χ_{θ} and the explicit formula for Φ_{θ}(f) is given in [16,§3] (see also [9]), namely

(5c.8.)  Φ_{θ}(f)(n) = ∫_{M∩B\B} f(bn)χ̄_{θ}(b) dḃ.

But then from [7] and (5c.8), we see we must show

(5c.9)  Φ_{θ}(q_{N,M}^{−1/2}(s)f(s^{−1} · s)) = q_{N,B}^{−1/2}(s)(Φ_{θ}f)(s^{−1} · s).

From (5c.8), the left side of (5c.9) is

q_{N,M}^{−1/2}(s) ∫_{M∩B\B} f(s^{−1}bns)χ̄_{θ}(b) dḃ
= q_{N,M}^{−1/2}(s)q_{B,M∩B}^{−1}(s) ∫_{M∩B\B} f(bs^{−1}ns)χ̄_{θ}(b) dḃ (S preserves 𝔟 and θ).

Thus proving (5c.9) comes down to demonstrating q_{N,M}^{−1/2}(s)q_{B,M∩B}^{−1}(s)q_{N,B}^{1/2}(s) ≡ 1. In

[19] it is shown that the latter equals q_{M,M∩B}^{−1/2}(s)q_{B,M∩B}^{−1/2}(s). Moreover, in the exponential solvable case one can choose 𝔟 so that this product is 1. I believe the same is true in general, but do not have an airtight proof. In any event, there may not be an S-invariant real polarization for θ — at best only a complex polarization, say 𝔠, which may vary with s ∈ S. Then there is an intertwining operator

γ_{𝔟,𝔠} : space of Ind_{B}^{N}χ_{θ} → space of 𝔠 − Ind_{N_{θ}}^{N}χ_{θ}.

A fundamental theorem in [1,Thm.III.3.1] asserts that the operator

γ_{𝔟,𝔠}^{−1} ∘ γ̃_{θ}(s) ∘ γ_{𝔟,𝔠}

does not depend on 𝔠. (Think of 𝔟 as fixed and 𝔠 as varying.) Therefore if we set

Φ_{θ}^{𝔠}(f) = γ_{𝔟,𝔠}^{−1} ∘ Φ_{θ}^{𝔠}(s) ∘ γ_{𝔟,𝔠},

and examine [7] again, we see that our argument is reduced to showing

(5c.10)  Φ_{θ}^{𝔠} (q_{N,M}^{−1/2}(s)f(s^{−1} · s)) = q_{N,D}^{−1/2}(s)(Φ_{θ}^{𝔠}f)(s^{−1} · s).

Moreover, it is enough to consider f ∈ D(N, M), the smooth, compactly supported (mod M) functions on N. In line with the previous remark, one should be able to do this by a close examination of the intertwining operators γ_{𝔟,𝔠} (see [7]). The actual argument we employ is induction on dim N — to which we at last proceed.

So now let 𝔷 = Cent 𝔫 and consider 𝔞 = 𝔷 ∩ 𝔪. Suppose dim 𝔞 > 0. Such an ideal will be S- and σ-invariant, and τ_{N} and γ_{θ} will be trivial on it. Moreover A ⊂ M and A ⊂ D. Thus the modulus for the action of S on A disappears from both sides of (5c.10). By induction, we may divide it out. So we may assume 𝔷 ∩ 𝔪 = {0}. Let 𝔞 = 𝔷 ∩ ker θ, again S- and σ-invariant, and suppose dim 𝔞 > 0. Consider the group N_{1} = N/A, p : N → N_{1}, the canonical projection. Set M_{1} = p(M) and note that M_{1} ≅ M. Of course γ_{θ} is trivial on A, but τ_{N} is not. Clearly we have

q_{N,M} = q_{N_{1},M}q_{A} q_{N,D} = q_{N_{1},D_{1}},

where D_{1} = p(D). But we can reason as follows. We have

Ind_{M}^{N}1 = Ind_{MA}^{N}Ind_{M}^{MA}1 = Ind_{MA}^{N} ∫ ⊕ ; Â χ dχ.

Moreover the intertwining operator for the latter direct integral decomposition is easily written down. Furthermore, the component at the place corresponding to χ = 1, which maps left M-invariant functions on N to left MA-invariant functions on N converts τ̃_{N}(s) into q_{A}^{−1/2}(s)τ̃_{N_{1}}(s). Thus if equation (5c.10) is proven for N_{1}, it is also true for N.

So we may assume 𝔷 ∩ 𝔪 = {0}, dim 𝔷 = 1 and θ|_{𝔷} ≠ 0. In that case, we consider the characteristic ideal 𝔞 = Cent [𝔫, 𝔫]. Either 𝔫 is Heisenberg, or 𝔞 is an abelian ideal strictly bigger than 𝔷. Set 𝔫_{1} = Cent_{𝔫}(θ|_{𝔞}), a σ- and S-invariant proper ideal in 𝔫. Set θ_{1} = θ|_{𝔫_{1}}. We consider two possibilities:

(a) 𝔫_{1} + 𝔪 ⫋ 𝔫 (b) 𝔫_{1} + 𝔪 = 𝔫.

(a) We will apply the induction assumption to the smaller algebra 𝔫_{0} = 𝔫_{1} + 𝔪 which is σ- and S- invariant. It contains both 𝔪 and 𝔡. The latter is because (from the Mackey Machine) we know γ_{θ} = Ind_{N_{1}}^{N}γ_{θ_{1}}. Hence M ⊂ N_{0} and D ⊂ N_{0}. Now τ_{N} = Ind_{N_{0}}^{N}Ind_{M}^{N_{0}}1. Write τ_{0} = Ind_{M}^{N_{0}}1. By the induction assumption, the result is true for N_{0} — i.e. (5c.10) holds with N replaced by N_{0} and f ∈ D(N_{0}, M). But the equivalence τ_{N} = Ind_{N_{0}}^{N}τ_{0} means in part that we can view any f ∈ L^{2}(N; M) as an element f_{0} ∈ L^{2}(N; L^{2}(N_{0}; M)) by f_{0}(n)(n_{0}) = f(n_{0}n). Combining this observation with the common factor q_{N,N_{0}}^{−1/2} on both sides, we see that (5c.10) also holds for N.

(b) In this case set M_{1} = M ∩ N_{1} = N_{1}^{σ}. So N_{1}/M_{1} is a symmetric space and dim N_{1} < dim N. Then N/M = N_{1}M/M ≈ N_{1}/M_{1}, and so q_{N,M} = q_{N_{1},M_{1}}. But by the Subgroup Theorem

τ_{N}|_{N_{1}} = Ind_{M}^{N}1|_{N_{1}} = Ind_{M_{1}}^{N_{1}}1 = τ_{N_{1}}.

Thus the left side of (5c.10) is unchanged when we view functions f ∈ D(N, M) as elements of D(N_{1}, M_{1}). By induction, that is the same as the right side with N replaced by N_{1}. But now we use the reasoning on the top of p. 103 in [7]. For any s ∈ S choose an s-invariant polarization 𝔠 which contains 𝔞. (This can be done since 𝔞 is clearly isotropic.) Then 𝔠 ⊂ (𝔫_{1})_{c} and is a polarization for θ_{1} — reflecting the fact that we still have γ_{θ} = Ind_{N_{1}}^{N}γ_{θ_{1}}. But when we view functions f ∈ D(N, M) as M_{1}-invariant functions on N_{1}, this introduces a factor of q_{N,N_{1}}^{−1/2}(s) into the formula for Φ_{ϕ}^{𝔠}f. Replacing q_{N_{1},D}^{−1/2} by q_{N,D}^{−1/2}q_{N,N_{1}}^{1/2}, we see the two factors involving q_{N,N_{1}}(s) cancel, and the result is proven for N.

We are reduced finally to the case that 𝔫 is Heisenberg. But then N/M can be symmetric (when 𝔷 ∩ 𝔪 = {0}) only when 𝔪/𝔷 is totally isotropic inside 𝔫/𝔷. This restricts S to be in a Borel subgroup of Sp(𝔫/𝔷). In that case, it leaves invariant some real polarization for any θ ≠ 0 on 𝔷 — namely 𝔪 itself. Therefore, the proof of Lemma 5.3 is done by the fact that, when M = B, the real polarization argument given earlier constitutes a proof.

Now we redo the Mackey Machine analysis of τ from §5b — but without the extraneous assumptions and with the incorporation of the Mackey obstructions as outlined above.

τ = Ind_{H}^{G}1 = Ind_{K}^{G}Ind_{H}^{K}1 = Ind_{K}^{G} ∫_{N̂(M)/S}^{⊕} Ind_{S_{γ}N}^{SN} χ̃_{θ}γ̃_{θ} × γ_{θ} dμ̇_{M}(γ_{θ})

= ∫_{N̂(M)/S}^{⊕} Ind_{S_{γ}N}^{G} χ̃_{θ}γ̃_{θ} × γ_{θ} dμ̇_{M}(γ_{θ})

= ∫_{N̂(M)/S}^{⊕} Ind_{R_{γ}N}^{G}Ind_{S_{γ}N}^{R_{γ}N} χ̃_{θ}γ̃_{θ} × γ_{θ} dμ̇_{M}(γ_{θ}).

So now we have to analyze

Ind_{S_{γ}N}^{R_{γ}N} χ̃_{θ}γ̃_{θ} × γ_{θ} = Ind_{K_{θ}N}^{G_{θ}N} χ̃_{θ}γ̃_{θ} × γ_{θ}

in analogy with (5b.1) and (5b.2).

Lemma 5.4. Suppose IndG̃_{θ} ; K̃_{θ} χ̃_{θ} is a direct integral

(5c.11)  IndG̃_{θ} ; K̃_{θ} χ̃_{θ} = ∫_{G̃_{θ}(K̃_{θ},χ̃_{θ})^{∧}}^{⊕} n_{θ}(ν)ν dμ_{θ}(ν).

Then

(5c.12)  Ind_{K_{θ}N}^{G_{θ}N} χ̃_{θ}γ̃_{θ} × γ_{θ} = ∫_{G̃_{θ}(K̃_{θ},χ̃_{θ})^{∧}}^{⊕} n_{θ}(ν)ν ⊗ γ̃_{θ} × γ_{θ} dμ_{θ}(ν);

and so

(5c.13)  Ind_{S_{γ}N}^{G} χ̃_{θ}γ̃_{θ} × γ_{θ} = ∫_{G̃_{θ}(K̃_{θ},χ̃_{θ})^{∧}}^{⊕} n_{θ}(ν)π_{θ,ν} dμ_{θ}(ν),

where

π_{θ,ν} = Ind_{G_{θ}N}^{G}ν ⊗ γ̃_{θ} × γ_{θ}.

Proof. It is enough to prove (5c.12) from which (5c.13) follows immediately. The induced representation on the left in (5c.12) acts in L^{2}(G_{θ}N; K_{θ}N; H_{γ}) by right translation. (We write L^{2}(A; B; H_{ω}) for the realization of the representation Ind_{B}^{A}ω on {f : A → H_{ω}, f(ba) = ω(b)f(a), ∫_{B\A} ||f||^{2} < ∞}.) The latter Hilbert space is naturally identified to L^{2}(G̃_{θ}N; K̃_{θ}N; H_{γ}). We then employ the unitary map given by restricting functions to G̃_{θ}, thereby obtaining the Hilbert space L^{2}(G̃_{θ}; K̃_{θ}; H_{γ}). G̃_{θ} still acts by right translation; but N now acts by

n · F (g̃_{θ}) = γ(g̃_{θ}ng̃_{θ}^{−1})F (g̃_{θ}).

Next we employ the unitary isomorphism specified by

(5c.14)  f ⊗ ξ → F_{f,ξ}(g_{θ}) = f(g_{θ})γ̃(g_{θ})ξ,

which converts the previous Hilbert space into L^{2}(G̃_{θ}; K̃_{θ}) ⊗ H_{γ}. It is straightforward to check that F_{f,ξ} satisfies the equivariance relation

F_{f,ξ}(k̃_{θ}g̃_{θ}) = (χ̃_{θ}γ̃_{θ})(k̃_{θ})F_{f,ξ}(g̃_{θ}).

Moreover, if we compute the transformed group action in the last Hilbert space we find that the N action is specified by

f ⊗ ξ → F_{f,ξ} → n · F_{f,ξ}(g̃_{θ}) = γ(g̃_{θ}ng̃_{θ}^{−1})F_{f,ξ}(g̃_{θ}) = γ(g̃_{θ})γ(n)f(g̃_{θ})ξ = F_{f,γ(n)ξ};

and the G̃_{θ}-action is determined by

f ⊗ ξ → F_{f,ξ} → F_{f,ξ}(g̃′_{θ}g̃_{θ}) = f(g̃′_{θ}g̃_{θ})γ̃(g̃′_{θ}g̃_{θ})ξ = f g̃_{θ}(g̃′_{θ}) ⊗ γ̃(g̃′_{θ})γ̃(g̃_{θ})ξ = F_{f g̃_{θ} ,γ(g̃_{θ})ξ}.

But this proves precisely that we have a unitary equivalence

Ind_{K̃_{θ}N}^{G̃_{θ}N}(χ̃_{θ}γ̃_{θ} × γ_{θ}) ≅ (IndG̃_{θ} ; K̃_{θ} χ̃_{θ}) ⊗ γ̃_{θ} × γ_{θ}

and the lemma is proven.

Now we put it all together to obtain

(5c.15)  τ = Ind_{H}^{G}1 = ∫_{N̂(M)/S}^{⊕} ∫_{G̃_{θ}(K̃_{θ},χ̃_{θ})^{∧}}^{⊕} n_{θ}(ν)π_{θ,ν} dμ_{θ}(ν)dμ̇_{M}(γ_{θ}).

Thus we have reduced the study of the symmetric space quasi-regular representation τ = Ind_{H}^{G}1 to the study of the induced representations IndG̃_{θ} ; K̃_{θ} χ̃_{θ}. This falls short of our goal

for two reasons: the latter are not quasi-regular representations (χ̃_{θ} ≠ 1); and also G̃_{θ}/K̃_{θ} is not symmetric. How can there be an inductive scheme if the inherited structure differs from the original — in two ways yet? Indeed, we shall overcome both difficulties with a single change of reference.

5d. Full generality. Set 𝔮_{θ} = ker θ|_{𝔫_{θ}}, Q_{θ} the corresponding analytic subgroup of N_{θ}. It is clear that [𝔤_{θ}, 𝔮_{θ}] ⊂ 𝔫_{θ}, and obviously θ[𝔤_{θ}, 𝔮_{θ}] = 0. That is, 𝔮_{θ} is an ideal in 𝔤_{θ}. Furthermore 𝔫_{θ}/𝔮_{θ} is central in 𝔤_{θ}/𝔮_{θ}. This is because [𝔤_{θ}, 𝔫_{θ}] ⊂ 𝔫_{θ} and θ[𝔤_{θ}, 𝔫_{θ}] = 0, so that [𝔤_{θ}, 𝔫_{θ}] ⊂ 𝔮_{θ}. In fact these properties lift to the groups. The reasoning follows [5,p. 207]. For g ∈ G_{θ} and X ∈ 𝔫_{θ}, it is clear that g · X − X ∈ 𝔮_{θ}. That is, the adjoint action of G_{θ} on 𝔫_{θ}/𝔮_{θ} is trivial. It follows that [G_{θ}, N_{θ}] ⊂ Q_{θ}. It is then further evident that [G̃_{θ}, Ñ_{θ}] ⊂ Q̃_{θ}, that is Ñ_{θ}/Q_{θ} is central in G̃_{θ}/Q_{θ}. (We note that the preceding idea originated in [1].)

We also know from Lemma 5.2 that K̃_{θ}/Q_{θ} = H̃_{θ} · Ñ_{θ}/Q_{θ}. Moreover, the character χ̃_{θ} satisfies σχ̃_{θ} = (χ̃_{θ})^{−}. Therefore, in place of considering the quasi-regular representation τ of the symmetric space G/H, we should consider the following scenario:

Z ⊂ Cent G ∩ N, connected and σ − invariant,

χ ∈ ĤZ, σχ = χ̄,
τ_{χ} = Ind_{HZ}^{G}χ.

If we redo the entire computation of section 5c starting with τ_{χ} in place of τ, the key point is that the inner integral in (5c.15) is still totally governed by (5c.11), i.e. by

τ_{θ} = IndG̃_{θ} ; K̃_{θ} χ̃_{θ} = Ind_{K̃_{θ}/Q_{θ}}^{G̃_{θ}/Q_{θ}} χ̃_{θ} = Ind_{H̃_{θ}N_{θ}/Q_{θ}}^{G̃_{θ}/Q_{θ}} χ̃_{θ}.

Since G̃_{θ}/H̃_{θ} = (G̃_{θ}/Q_{θ})/(H̃_{θ}/Q_{θ}) is symmetric, Ñ_{θ}/Q_{θ} is central in G̃_{θ}/Q_{θ}, and σχ̃_{θ} = (χ̃_{θ})^{−}, the structure of the little homogeneous space representation τ_{θ} is an exact replica of that of τ_{χ}. The induction scheme is now completely practical and the final result is

Theorem 5.5. Let G be algebraic, σ an involution, N the unipotent radical of G, R a σ-invariant Levi factor. Set H = G^{σ}, S = R^{σ}, M = N^{σ}, and let Z be a connected central unipotent σ-invariant subgroup of G together with a character χ ∈ ĤZ satisfying σχ = χ̄. Let ψ ∈ 𝔤^{∗} be any functional satisfying σψ = −ψ and ψ|_{𝔥+𝔷} = \frac{1}{i}dχ. Suppose that for θ ∈ ψ + (𝔪 + 𝔷)^{⊥} we have the direct integral decomposition

(5d.1)  IndG̃_{θ} ; K̃_{θ} χ̃_{θ} = Ind_{K̃_{θ}/Q_{θ}}^{G̃_{θ}/Q_{θ}} χ̃_{θ} = ∫_{G̃_{θ}(K̃_{θ},χ̃_{θ})^{∧}}^{⊕} n_{θ}(ν)ν dμ_{θ}(ν).

Then we also have

τ_{χ} = Ind_{HZ}^{G}χ

= ∫_{[ψ+(𝔪+𝔷)^{⊥}]/H}^{⊕} ∫_{G̃_{θ}(K̃_{θ},χ̃_{θ})^{∧}}^{⊕} n_{θ}(ν)π_{θ,ν} dμ_{θ}(ν)dθ̇ (5d.2)

= ∫_{G·[ψ+(𝔪+𝔷)^{⊥}]/G}^{⊕} n_{θ} ∫_{G̃_{θ}(K̃_{θ},χ̃_{θ})^{∧}}^{⊕} n_{θ}(ν)π_{θ,ν} dμ_{θ}(ν)dθ̈, (5d.3) where n_{θ} = #{G · θ ∩ [ψ + (𝔪 + 𝔷)^{⊥})]}/H.

Proof. In fact there is not much more to do beyond what we have already done. We only need to review and redo the computations in section 5c, and make some observations:

(1) The set ψ + (𝔪 + 𝔷)^{⊥} is independent of the choice of ψ.

(2) ψ + (𝔪 + 𝔷)^{⊥} ⊂ 𝔪^{⊥}.

(3) For any θ ∈ ψ + (𝔪 + 𝔷)^{⊥}, we have σθ = −θ.

(4) The representation Ind_{MZ}^{N}χ is still multiplicity-free because if θ ∈ ψ + (𝔪 + 𝔷)^{⊥}, then N · θ ∩ [ψ + (𝔪 + 𝔷)^{⊥}] = M · θ. Indeed if n · θ ∈ ψ + (𝔪 + 𝔷)^{⊥}, then since ψ + (𝔪 + 𝔷)^{⊥} ⊂ 𝔪^{⊥}, we have n · θ ∈ 𝔪^{⊥} ⇒ n · θ = m · θ for some m ∈ M (since N/M is symmetric).

(5) The word-for-word analog of the argument in [19,Thm.5.3] shows that: If θ_{1}, θ_{2} ∈ ψ+(𝔪+𝔷)^{⊥} are G-conjugate, say g·θ_{1} = θ_{2}, then (replacing g by σ(g) if necessary), we have g · H_{θ_{1}} = H_{θ_{2}}. Since N is characteristic, we also have g · N_{θ_{1}} = N_{θ_{2}}. Therefore K_{θ_{1}} and K_{θ_{2}} are g-conjugate. Furthermore, the numbers n_{θ} must be finite.

(6) The character χ must be of order two on H — since on H, σχ = χ̄ = χ ⇒ χ^{2} = 1. In particular χ(H^{0}) = 1.

For the record, here is the formal derivation of the direct integral decomposition:

τ_{χ} = Ind_{HZ}^{G}χ

= Ind_{K}^{G}Ind_{HZ}^{K}χ

= Ind_{K}^{G}Ind_{SMZ}^{SN}χ

= Ind_{K}^{G} ∫_{[ψ+(𝔪+𝔷)^{⊥}]/H}^{⊕} Ind_{K_{θ}N}^{K} χ̃_{θ}γ̃_{θ} × γ_{θ} dθ̇

(from Ind_{MZ}^{N}χ = ∫_{[ψ+(𝔪+𝔷)^{⊥}]/H}^{⊕} γ_{θ} dθ̇ (by [6]) and the
evident (and easily proven) analog of Lemma 5.3))

= ∫_{[ψ+(𝔪+𝔷)^{⊥}]/H}^{⊕} Ind_{K_{θ}N}^{G} χ̃_{θ}γ̃_{θ} × γ_{θ} dθ̇

= ∫_{[ψ+(𝔪+𝔷)^{⊥}]/H}^{⊕} Ind_{G_{θ}N}^{G}Ind_{K_{θ}N}^{G_{θ}N} χ̃_{θ}γ̃_{θ} × γ_{θ} dθ̇

= ∫_{[ψ+(𝔪+𝔷)^{⊥}]/H}^{⊕} ∫_{G̃_{θ}(K̃_{θ},χ̃_{θ})^{∧}}^{⊕} Ind_{G_{θ}N}^{G}n_{θ}(ν)ν ⊗ γ̃_{θ} × γ_{θ} dμ_{θ}(ν)dθ̇ (Lemma 5.4)

= ∫_{[ψ+(𝔪+𝔷)^{⊥}]/H}^{⊕} ∫_{G̃_{θ}(K̃_{θ},χ̃_{θ})^{∧}}^{⊕} n_{θ}(ν)π_{θ,ν} dμ_{θ}(ν)dθ̇

= ∫_{G·[ψ+(𝔪+𝔷)^{⊥}]/G}^{⊕} n_{θ} ∫_{G̃_{θ}(K̃_{θ},χ̃_{θ})^{∧}}^{⊕} n_{θ}(ν)π_{θ,ν} dμ_{θ}(ν)dθ̈.

Formulas (5d.1-3) embody the inductive scheme that we have envisioned. We use them critically in the next two sections.

Remark. We note that the entire scheme of this section (and so its consequences in the next two sections) relies critically on G being algebraic. Without that, there is no Levi decomposition and one cannot even get started in setting up an inductive technique.

6. The orbital criterion. We reformulate Theorem 4.1 so as to accommodate the more general representation τ_{χ} in place of the quasi-regular representation τ. We prove the reformulated form of the theorem using the inductive scheme from §5 (see Theorem 5.5). Then we obtain Theorem 4.1 as a special case. In fact, it is quite clear how the orbital characterizations of Theorem 4.1 should generalize to τ_{χ}. The result is

Theorem 6.1. Let G be algebraic, σ an involution, Z a connected central unipotent σinvariant subgroup with a character χ ∈ ĤZ satisfying σχ = χ̄. Select ψ as in Theorem 5.5. Then the representation τ_{χ} = Ind_{HZ}^{G}χ is

(i) irreducible iff there is a Zariski-open H-orbit on ψ + (𝔥 + 𝔷)^{⊥}, and for any ϕ in that orbit, G_{ϕ} ⊂ HZ;

(ii) a finite sum of irreducibles iff there is a finite family of open H-orbits in ψ+(𝔥+𝔷)^{⊥} whose union is conull.

Remark. It is obvious that Theorem 4.1 is a special case of Theorem 6.1 (taking Z = {1}, χ = 1). As in that theorem, either condition in (i) forces Cent G ⊂ HZ.

Proof. First we make an important clarification on notation. In the preceding, all uses of the perp symbol (⊥) have been self-evident — namely, 𝔥^{⊥} is in 𝔤^{∗}, 𝔪^{⊥} and (𝔪 + 𝔷)^{⊥} are in 𝔫^{∗}, and U^{⊥} is in V̂. But in this section and the next we shall have to distinguish more closely the set in which the perps are taken. We use the same device as in [12,§2] — namely, if 𝔞 ⊂ 𝔟 we set 𝔞^{⊥}(𝔟) = {ϕ ∈ 𝔟^{∗} : ϕ(𝔞) = 0}. Then if 𝔞 ⊂ 𝔟 ⊂ 𝔠, the symbols 𝔞^{⊥}(𝔟) and 𝔞^{⊥}(𝔠) are unambiguous.

Now suppose there is a Zariski-open H-orbit O on ψ + (𝔥 + 𝔷)^{⊥}(𝔤) for which G_{ϕ} ⊂ HZ if ϕ ∈ O. The projection p : 𝔤^{∗} → 𝔫^{∗}, ϕ → ϕ|_{𝔫} supplies us with a Zariski-open H-orbit on ψ + (𝔪 + 𝔷)^{⊥}(𝔫). (We abuse notation slightly by using ψ for ψ|_{𝔫}.) Therefore measuretheoretically, the set [ψ+(𝔪+𝔷)^{⊥}(𝔫)]/H is a singleton. Hence the outer integral in formula (5d.2) reduces to an atom. Next we show that the same hypotheses apply to the inner integral, or little homogeneous spaces in (5d.1). Let ϕ ∈ O, θ = ϕ|_{𝔫}. Consider O ∩ p^{−1}(θ), Zariski-open and non-empty in p^{−1}(θ). Let ϕ_{1}, ϕ_{2} ∈ O ∩ p^{−1}(θ). By hypothesis there is h ∈ H such that h · ϕ_{1} = ϕ_{2}. Restricting to 𝔫, we see that h ∈ H_{θ}. That is, H_{θ} has a single orbit on O ∩ p^{−1}(θ). Then project

p^{−1}(θ) → θ + 𝔨_{θ}^{⊥}(𝔤_{θ}).

The image of O∩p^{−1}(θ) will be a Zariski-open subset which is a single H_{θ}-orbit. Moreover, if ξ is in that image, it is classical (see [1,Prop.II.1.3]) that (G_{θ})_{ξ} = G_{ϕ}N_{θ}. Therefore (G_{θ})_{ξ}/Q_{θ} = G_{ϕ}N_{θ}/Q_{θ} ⊂ (HZ)_{ϕ}N_{θ}/Q_{θ} ⊂ H_{θ}N_{θ}/Q_{θ}. Hence, we may apply an induction hypothesis to obtain irreducibility in (5d.1), and so also for the inner integral in (5d.2). From Theorem 5.5 we obtain the irreducibility of τ_{χ}.

If we moderate the assumption to the condition in statement (ii), then essentially the same argument shows that [ψ + (𝔪 + 𝔷)^{⊥}(𝔫)]/H is finite and n_{θ} < ∞. The induction hypothesis applies to the inner integral again, and we get a finite sum of irreducibles. Combining these observations, we see that τ_{χ} is also a finite sum of irreducibles.

This completes the proof of the sufficiency of the orbit condition in statements (i) and (ii). In order to prove the converse, we require another Lemma 6.2. Let ϕ ∈ ψ + (𝔥 + 𝔷)^{⊥}(𝔤), θ = ϕ|_{𝔫}. Then

M_{θ} · ϕ = ϕ + (𝔤_{θ} + 𝔫 + 𝔥)^{⊥}(𝔤).

Remark. This generalizes the classical ([15,Lemma 2.4] [22,Lemma 2]), and oft-used, fact that

N_{θ} · ϕ = ϕ + (𝔤_{θ} + 𝔫)^{⊥}(𝔤).

Proof. First 𝔪_{θ} · ϕ ∈ (𝔤_{θ} + 𝔫 + 𝔥)^{⊥} because

ϕ[𝔤_{θ}, 𝔪_{θ}] ⊂ θ[𝔤_{θ}, 𝔫] = 0,

ϕ[𝔪_{θ}, 𝔫] ⊂ θ[𝔪_{θ}, 𝔫] = 0,

\begin{aligned}ϕ[𝔪_{θ}, 𝔥] & = θ[𝔪_{θ}, 𝔪] + ϕ[𝔪_{θ}, 𝔰] \\ & = ϕ[𝔪_{θ}, 𝔰] ⊂ θ(𝔪) = 0. \end{aligned}

Thus M_{θ} · ϕ ⊂ ϕ + (𝔤_{θ} + 𝔫 + 𝔥)^{⊥}(𝔤). But since M is unipotent its orbits are closed and connected. So it is enough to show equal dimension, that is to show

(6.1)  dim 𝔪_{θ}/𝔪_{ϕ} = dim 𝔤 − dim(𝔤_{θ} + 𝔫 + 𝔥).

Proof of (6.1). The right side has dimension

dim 𝔤 − [dim 𝔤_{θ} + dim 𝔫 + dim 𝔥 − dim 𝔤_{θ} ∩ 𝔫 − dim 𝔤_{θ} ∩ 𝔥 − dim 𝔫 ∩ 𝔥 + dim 𝔤_{θ} ∩ 𝔫 ∩ 𝔥] = dim 𝔤 − dim 𝔤_{θ} − dim 𝔫 − dim 𝔥 + dim 𝔫_{θ} + dim 𝔥_{θ} + dim 𝔪 − dim 𝔪_{θ}.

To compute the dimension of the left side of (6.1) we observe that (using p : 𝔫^{∗} → 𝔪^{∗} for the projection)

\begin{aligned}𝔪_{ϕ} & = \{X ∈ 𝔪 : ϕ[𝔤, X] = 0\} = \{X ∈ 𝔪 : θ[𝔤, X] = 0\} \\ & = \{X ∈ 𝔪 : 𝔤 · θ(X) = 0\} = (p(𝔤 · θ))^{⊥}(𝔪). \end{aligned}

Therefore

dim 𝔪_{ϕ} = dim 𝔪 − dim p(𝔤 · θ) = dim 𝔪 − [dim 𝔤 · θ − dim(𝔤 · θ ∩ 𝔪^{⊥})(𝔫)]
= dim 𝔪 − dim 𝔤 · θ + dim 𝔥 · θ.

Similarly we have 𝔪_{θ} = (p(𝔫 · θ))^{⊥}(𝔪), which leads to

dim 𝔪_{θ} = dim 𝔪 − dim 𝔫 · θ + dim 𝔪 · θ.

Combining, we obtain

dim 𝔪_{θ}/𝔪_{ϕ} = dim 𝔪 · θ + dim 𝔤 · θ − dim 𝔥 · θ − dim 𝔫 · θ

which is the same as the right side of (6.1).

Now, proceeding to the proof of the necessity of the orbital condition in (i), suppose that τ_{χ} is irreducible. Then from (5d.2), [ψ + (𝔪 + 𝔷)^{⊥}(𝔫)]/H must be a singleton. In particular, H has a Zariski-open orbit on ψ + (𝔪 + 𝔷)^{⊥}(𝔫). Also from (5d.1-2), IndG̃_{θ} ; K̃_{θ} χ̃_{θ} must be irreducible. Apply an induction hypothesis. The conclusion is that K_{θ} — and so H_{θ} — has a Zariski-open orbit on θ + 𝔨_{θ}^{⊥}(𝔤_{θ}), and for ξ in that orbit (G_{θ})_{ξ} ⊂ K_{θ}. But K_{θ} = H_{θ}N_{θ} and (G_{θ})_{ξ} = G_{ϕ}N_{θ}. Moreover, Corollary 4.4 generalizes easily to the fact that for any ϕ in an open H-orbit on ψ + (𝔪 + 𝔷)^{⊥}, we have G_{ϕ}^{0} = H_{ϕ}^{0}Z. Since Z ⊂ N_{θ} and N_{θ} is simply connected, it is clear that G_{ϕ} ⊂ HZ. Therefore, it remains to show that there is a Zariski-open orbit. But this is a simple consequence of three facts that we have at our disposal, namely: (i) H_{θ} has a Zariski-open orbit on θ + 𝔨_{θ}^{⊥}(𝔤_{θ}); (ii) M_{θ} · ϕ = ϕ + (𝔤_{θ} + 𝔫 + 𝔥)^{⊥}(𝔤) (Lemma 6.2); and (iii) H has a Zariski-open orbit on ψ + (𝔪 + 𝔷)^{⊥}(𝔫).

We leave the details of the similar — but much easier — argument for necessity in item (ii) to the reader.

7. Algebraic symmetric spaces have finite multiplicity. We now reap an enormous benefit from the inductive scheme set up in §5. Namely, we settle the conjecture from [18] — at least in the algebraic case. We prove that algebraic symmetric spaces always have finite multiplicity. In fact we prove the slightly more general

Theorem 7.1. Let G be algebraic, σ an involution, H = G^{σ}, Z a connected unipotent central σ-invariant subgroup, with χ ∈ ĤZ satisfying σχ = χ̄. Then the representation

τ_{χ} = Ind_{HZ}^{G}χ

has finite multiplicity.

Of course we have as a corollary (by taking Z = {1}, χ = 1)

Theorem 7.2. Let G be algebraic, σ an involution, H = G^{σ}. Then the quasi-regular rep τ = Ind_{H}^{G}1 has finite multiplicity.

Proof of Theorem 7.1. The proof, which is by induction on dimension, makes critical use of formula (5d.3) from Theorem 5.5. The key point is that the “little representations”

τ_{θ} = Ind_{K̃_{θ}/Q_{θ}}^{G̃_{θ}/Q_{θ}} χ̃_{θ}

have the same structural properties as τ_{χ}. Now the result is patently true for groups of dimension 1. To implement the induction argument, we observe that the only way the numbers dim G̃_{θ}/Q_{θ} could fail to be smaller than dim G is if Q_{θ} is trivial and G = G_{θ}. But then N = N_{θ} which implies γ_{θ} is trivial on the commutator subgroup of N — that is, it is a character. But then if Q_{θ} is trivial, N must be 1-dimensional and G is reductive. The structure then devolves to

τ_{χ} = Ind_{H}^{G}χ, σχ = χ̄.

But we have seen that this forces χ to be trivial on H^{0}, and the finite multiplicity of τ_{χ} follows from that of

Ind_{H^{0}}^{G}1,

which fact is proven in [2].

Therefore by (5d.3) and the induction assumption, to prove finite multiplicity it is enough to prove that the numbers n_{θ} are finite. The proof of that fact is almost the same argument as in [18], which is the heuristic argument motivating the original finite multiplicity conjecture. Specifically, we must show that generically on ψ + (𝔪 + 𝔷)^{⊥}(𝔫) the number of H-orbits is finite. But we have

dim H · θ ≤ dim G · θ ∩ [ψ + (𝔪 + 𝔷)^{⊥}(𝔫)]
≤ dim 𝔤 · θ ∩ (𝔪 + 𝔷)^{⊥}(𝔫)

(7.1) = dim 𝔥 · θ (see below)

= dim H · θ.

That is H has open orbits on ψ + (𝔪 + 𝔷)^{⊥}(𝔫), and by the algebraic assumption there can only be finitely many.

Proof of (7.1). It suffices to show: θ ∈ 𝔪^{⊥} and X · θ ∈ 𝔪^{⊥}, X ∈ 𝔤 ⇒ X · θ ∈ 𝔥 · θ. Write X ∈ 𝔤 uniquely as X = Y +W, Y ∈ 𝔥, W ∈ 𝔮. Then, since σ(θ) = −θ and σ(X ·θ) = −X ·θ, we have (Y −W)·−θ = σ(X)·σθ = σ(X ·θ) = −X ·θ = −(Y +W)·θ. Hence W ·θ = 0. □

Now I consider the possible boundedness of the finite multiplicity in τ_{χ} or in the quasiregular representation τ = Ind_{H}^{G}1. The heuristic argument in [18] was also evidence for the boundedness (in addition to the finiteness) of the multiplicity. The argument made use of the fact that the Orbit Method suggested that the multiplicities were intimately related to the numbers

n_{ϕ} = #(G · ϕ ∩ 𝔥^{⊥})/H, ϕ ∈ 𝔥^{⊥}.

That these numbers are finite for symmetric spaces G/H was in fact proven in [19,Thm.5.3]. Of course n_{ϕ} ≤ # components of (G · ϕ ∩ 𝔥^{⊥}), and then a simple application of [17,in the special case k = 0] shows that

n_{G,H} := sup_{ϕ∈𝔥^{⊥}} n_{ϕ} ≤ sup_{ϕ∈𝔥^{⊥}} # components of (G · ϕ ∩ 𝔥^{⊥}) < ∞.

We can now state

Theorem 7.3. Suppose that for any reductive symmetric space G/H the multiplicity function in τ_{χ} = Ind_{H}^{G}χ, σχ = χ̄ is bounded above by n_{G,H}. Then for any algebraic symmetric space the multiplicity function in τ_{χ} (Theorem 7.2) or τ (Theorem 7.1) is uniformly bounded.

Proof. We incorporate the twist into the argument. In fact the induction assumption is precisely that

multiplicity in τ_{χ} ≤ sup_{ϕ∈ψ+(𝔥+𝔷)^{⊥}(𝔤)} #{G · ϕ ∩ [ψ + (𝔥 + 𝔷)^{⊥}(𝔤)]}/H.

In analogy with the proof of Theorem 7.2, the reductive case comes down to precisely that the multiplicity is dominated by n_{G,H} — which unfortunately we can only postulate at this point. As usual we use (5d.3). Set

(𝔤^{∗}/G, 𝔥, ψ) = set of G-orbits in 𝔤^{∗} that meet ψ + (𝔥 + 𝔷)^{⊥}(𝔤)
(𝔫^{∗}/G, 𝔪, ψ) = set of G-orbits in 𝔫^{∗} that meet ψ + (𝔪 + 𝔷)^{⊥}(𝔫)
(𝔤_{θ}^{∗}/G̃_{θ}, 𝔥_{θ}, θ) = set of G̃_{θ}-orbits in 𝔤_{θ}^{∗} that meet θ + 𝔨_{θ}^{⊥}(𝔤_{θ}).

Then by (5d.3) and the induction hypothesis, the theorem will be proven if we can show that — for ϕ ∈ ψ + (𝔥 + 𝔷)^{⊥}, θ = ϕ|_{𝔫}, ξ = ϕ|_{𝔤_{θ}}, corresponding to which

G · ϕ ∈ (𝔤^{∗}/G, 𝔥, ψ), G · θ ∈ (𝔫^{∗}/G, 𝔪, ψ), G̃_{θ} · ξ ∈ (𝔤_{θ}^{∗}/G̃_{θ}, 𝔥_{θ}, θ)

— we have

(7.2)  #{G · ϕ ∩ [ψ + (𝔥 + 𝔷)^{⊥}(𝔤)]}/H = n_{θ} · #{G̃_{θ} · ξ ∩ [θ + 𝔨_{θ}^{⊥}(𝔤_{θ})]}/H̃_{θ}.

Consider first the surjective map

p : Ω ∈ (𝔤^{∗}/G, 𝔥, ψ) → ω ∈ (𝔫^{∗}/G, 𝔪, ψ)
p(ϕ) = θ = ϕ|_{𝔫}.

For fixed θ ∈ 𝔪^{⊥}, we show there is a bijection between the sets

(𝔤_{θ}^{∗}/G_{θ}, 𝔥_{θ}, θ) and {Ω ∈ (𝔤^{∗}/G, 𝔥, ψ) : θ ∈ p(Ω)}.

Let ξ ∈ 𝔤_{θ}^{∗}, ξ(𝔥_{θ}) = 0, ξ|_{𝔫_{θ}} = θ|_{𝔫_{θ}}. Define Ω_{ξ} = G · ϕ where ϕ is any extension of ξ + θ that is 0 on 𝔥. This is well-defined as follows: if ξ_{1} = g_{1} · ξ, g_{1} ∈ G_{θ}, ξ_{1}(𝔥_{θ}) = 0, then let ϕ_{1} be any extension of ξ_{1} + θ that is 0 on 𝔥. Then g_{1} · ϕ|_{𝔫} = θ = ϕ_{1}|_{𝔫}, g_{1} · ϕ|_{𝔤_{θ}} = ξ_{1} = ϕ_{1}|_{𝔤_{θ}} ⇒ g_{1} · ϕ = n · ϕ_{1} some n ∈ N_{θ} ⇒ G · ϕ = G · ϕ_{1}. Claim: the map T : G_{θ} · ξ → Ω_{ξ} is a bijection.

Onto: Let ϕ ∈ 𝔤^{∗}, ϕ(𝔥) = 0, ϕ|_{𝔫} = θ. Let ξ = ϕ|_{𝔤_{θ}}. Then ξ|_{𝔫_{θ}} = θ|_{𝔫_{θ}} and ξ(𝔥_{θ}) = ϕ(𝔥_{θ}) ⊂ ϕ(𝔥) = 0.

1-1: Suppose ξ_{1}, ξ_{2} ∈ 𝔤_{θ}^{∗}, ξ_{j}|_{𝔫_{θ}} = θ|_{𝔫_{θ}}, ξ_{j}(𝔥_{θ}) = 0. Let ϕ_{1}, ϕ_{2} be extensions and suppose g · ϕ_{1} = ϕ_{2}. Then g · θ = θ ⇒ g ∈ G_{θ}. But then g · ξ_{1} = ξ_{2} for g ∈ G_{θ}.

Now fix a G-orbit Ω = G · ϕ that meets ψ + (𝔥 + 𝔷)^{⊥}(𝔤). It maps to a G-orbit ω = G · θ that meets ψ + (𝔪 + 𝔷)^{⊥}(𝔫). Moreover it is clear that H-orbits in the former map to H-orbits in the latter. So we get a surjective map

{G · ϕ ∩ [ψ + (𝔥 + 𝔷)^{⊥}(𝔤)]}/H → {G · θ ∩ [ψ + (𝔪 + 𝔷)^{⊥}(𝔫)]}/H.

In general the map fails to be injective. Now there is a unique orbit Λ = G_{θ} · ξ in (𝔤_{θ}^{∗}/G_{θ}, 𝔥_{θ}, θ) that maps to Ω under the map T. Fix an H-orbit in ω ∩ [ψ + (𝔪 + 𝔷)^{⊥}(𝔫)]. Let C_{1}, …, C_{r} be the H-orbits in Ω ∩ [ψ + (𝔥 + 𝔷)^{⊥}(𝔤)] that map to it. Let C′_{1}, …, C′_{s} be the H_{θ}-orbits in Λ ∩ (θ + k_{θ}^{⊥}(𝔤_{θ}). We need to show: r = s and the H-orbits correspond under the map T.

So let

ϕ = ϕ_{1}, ϕ_{2}, …, ϕ_{r} be in distinct H-orbits in G · ϕ ∩ [ψ + (𝔥 + 𝔷)^{⊥}(𝔤)]

while

θ = θ_{1}, θ_{2}, …, θ_{r} are in the same H-orbit in G · θ ∩ [ψ + (𝔪 + 𝔷)^{⊥}(𝔫)].

We show there are exactly r distinct H_{θ}-orbits in G_{θ} · ξ ∩ [θ + 𝔨_{θ}^{⊥}]. First, there exists at least r orbits. To see that, choose g_{j} ∈ G so that ϕ_{j} = g_{j} · ϕ. Then ∃h_{j} ∋ h_{j} · θ = g_{j} · θ (because all θ_{j} are in the same H-orbit). Then h_{j}^{−1}g_{j} ∈ G_{θ}. Let ξ_{j} = h_{j}^{−1}g_{j} · ξ. Claim: these are all in distinct H_{θ}-orbits in G_{θ} ·ξ ∩[θ +𝔨_{θ}^{⊥}]. If not, we have h_{θ}h_{1}^{−1}g_{1} ·ξ = h_{2}^{−1}g_{2} ·ξ. Then h_{θ}h_{1}^{−1}g_{1} · ϕ and h_{2}^{−1}g_{2} · ϕ have the same restriction to 𝔫 and 𝔤_{θ}, and both vanish on 𝔥. Then we use M_{θ} · ϕ = ϕ + (𝔤_{θ} + 𝔫 + 𝔥)^{⊥}(𝔤) to get h_{θ}h_{1}^{−1}g_{1} · ϕ = m_{θ}h_{2}^{−1}g_{2} · ϕ ⇒ ϕ_{1}, ϕ_{2} are in the same H-orbit, a contradiction.

Now why can’t there be more than r orbits? Suppose ξ_{1}, …, ξ_{s}, s ≥ r, lie in distinct H_{θ}-orbits in G_{θ} · ξ ∩ [θ + 𝔨_{θ}^{⊥}]. Take ϕ′_{j} any extension of ξ_{j} + θ which is 0 on 𝔥. These lie in distinct H-orbits because if h · ϕ′_{1} = ϕ′_{2} ⇒ h ∈ H_{θ}, contradicting the fact that the ξ_{j} lie in distinct H_{θ}-orbits. But they all have the same restriction to 𝔫, contradicting that there are exactly r distinct H-orbits in G · ϕ ∩ [ψ + (𝔥 + 𝔷)^{⊥}(𝔤)] lying over the single H-orbit in G · θ ∩ [ψ + (𝔪 + 𝔷)^{⊥}(𝔫)].

Finally it is clear from the preceding paragraph that the map T : Λ ∩ (θ + 𝔨_{θ}^{⊥}) → Ω ∩ [ψ + (𝔥 + 𝔷)^{⊥}] is a bijection of the H_{θ}-orbits in the former to the H-orbits in the latter. □

8. Another example and two more questions. The point of this concluding section is to demonstrate that (quasi-)orbital symmetric spaces are actually quite common; and to raise two questions on orbital symmetric spaces that I have not been able to settle.

Consider the semidirect product G_{n} = GL(n, ℝ) ⋉ ℝ^{n} with the natural action. Identify G_{n} to a subgroup of GL(n + 1, ℝ) in the usual way

G_{n} = \{( \begin{matrix}r & v \\ 0 & 1 \end{matrix} ) : r ∈ GL(n, ℝ), v ∈ ℝ^{n×1}, \} .

Let ϵ = (ϵ_{1}, …, ϵ_{n}; 1) denote a diagonal matrix in GL(n + 1, ℝ) with ϵ_{j} = ±1, 1 ≤ j ≤ n. Define an involution σ_{ϵ} on G_{n} by σ_{ϵ}(g) = ϵgϵ. Note that any standard parabolic subgroup of GL(n + 1, ℝ) is σ_{ϵ}-invariant. In particular the maximal parabolic G_{n} is σ_{ϵ}-invariant.

Proposition 8.1. For any ϵ, the symmetric space G_{n}/G_{n}^{σ_{ϵ}} is orbital.

Proof. First note that it is no loss of generality to assume ϵ is of the form

(8.1)  ϵ = (+1, …, +1, −1, …, −1; 1),

that is r plus ones followed by s minus ones, r + s = n. Indeed, given any ϵ we pick a representative w of the element in the Weyl group (in GL(n + 1, ℝ)) that conjugates the original ϵ into the one of the form (8.1). But since the last diagonal entry is +1, w is actually in GL(n, ℝ) ⊂ G_{n}.

Now it is easy to see that with ϵ given by (8.1), we have

H = G_{n}^{σ_{ϵ}} = \{( \begin{matrix}A & 0 & x \\ 0 & D & 0 \\ 0 & 0 & 1 \end{matrix} ) : A ∈ GL(r, ℝ), D ∈ GL(s, ℝ), x ∈ ℝ^{r×1}\} ≅ G_{r} × GL(s, ℝ).

If s = 0, then G_{n}^{σ_{ϵ}} = G_{n} and there is nothing to do. If r = 0, we are in the case of an abelian symmetric space with a Zariski-open orbit and the result follows from [11]. Otherwise, the set Û(V) is naturally identified to

\{( \begin{matrix}I & 0 & 0 \\ 0 & I & 0 \\ 0 & y & 1 \end{matrix} ) , y ∈ ℝ^{1×s}\}

on which S = ( \begin{matrix}A & 0 & 0 \\ 0 & D & 0 \\ 0 & 0 & 1 \end{matrix} ) has a Zariski-open orbit. Furthermore, for y = (0, …, 0, 1),

the stability group in R is ≅ G_{n−1} and the little homogeneous space is isomorphic to

G_{n−1}/G_{n−1}^{σ_{ϵ′}}, ϵ′ = (−1_{1}, …, −1_{r}, +1_{1}, …, +1_{s−1}; 1).

But the latter is conjugate-isomorphic to the symmetric space G_{n−1}/G_{n−1}^{σ_{ϵ′′}}, where ϵ′′ = (+1_{1}, …, +1_{s−1}, −1_{1}, …, −1_{r}; 1). Applying an induction assumption, we see that eventually we are reduced to an abelian symmetric space with a Zariski-open orbit. □

One could play the same game over ℂ, or with SL(n, ·) instead of GL(n, ·), and with other split groups. I have a great deal of computational evidence for the general result suggested by Proposition 8.1 — although at the moment, no complete proof. So I formulate

Question 8.2. Let G be a split connected reductive Lie group, σ an involution of G which is not a Cartan involution. Let P be any parabolic subgroup of G which is σ-invariant. Then is it true that P/P^{σ} is quasi-orbital?

Here is the second unresolved issue. Suppose τ = Ind_{H}^{G}1 is orbital. This says that τ is monomial in a very strong sense. The question is whether the monomial nature of τ is accidental or intrinsic — that is, if an irreducible arises from an orbital symmetric space, then must its Mackey Machine realization also be monomial? The latter can be examined through the lens of Theorem 5.5

τ = ∫_{N̂(M)/S}^{⊕} Ind_{K̃_{θ}N}^{G} χ̃_{θ}γ̃_{θ} × γ_{θ}.

The irreducibility of τ forces S to have a Zariski-open orbit on N̂(M) and IndG̃_{θ} ; K̃_{θ} χ̃_{θ} to be irreducible. Applying the inductive scheme, we would deduce that the latter is “Mackeymonomial”; and then to conclude the same for τ it would be enough to have a positive resolution of

Question 8.3. Suppose that H has a Zariski-open orbit on 𝔪^{⊥}. Does it follow that there exists a G_{θ}-invariant real polarization for θ?

9. Appendix. Here is a proof, supplied by the referee, of the fact that if τ = Ind_{H}^{G}1 is irreducible, then D(G/H) is trivial. First we observe that if L ∈ D(G/H), then its formal adjoint L^{∗} (on C_{c}^{∞}(G/H)) is also in D(G/H). Thus the problem reduces to showing that a formally self-adjoint operator in D(G/H) must be scalar. This conclusion is a consequence of the following two facts:

(i) L|_{C_{c}^{∞}(G/H)} is essentially self-adjoint (ii) τ(G) commutes with the spectral projections of the self-adjoint extension L̄ of

L|_{C_{c}^{∞}(G/H)}.

Point (ii) is fairly standard — cf. Kato, Perturbation Theory for Linear Operators, Chapter 6, Section 5 (1966 ed). But point (i) is delicate. It follows from a result due to E. Thomas which is found in the article Commutativity of invariant differential operators on nilpotent homogeneous spaces K\G with finite multiplicity, Comm. Pure & Applied Math., vol. 45 (1992), 681-748 (see Theorem 7.2 in the preprint version of the article for a fuller discussion). That result asserts that: if τ is a type I representation of a Lie group G whose multiplicities are finite, and if L is a closed symmetric operator on a dense domain D ⊂ H_{τ} satisfying

(a) τ(g)D = D, ∀g ∈ G

(b) τ(g)Lv = Lτ(g)v, ∀g ∈ G, ∀v ∈ D, then L is self-adjoint. Thomas’s result applies to an irreducible quasi-regular representation τ since such a representation has a fortiori finite multiplicity.

References

1. L. Auslander & B. Kostant, Polarization and unitary representations of solvable Lie groups, Inventiones Math. 14 (1971), 255-254.

2. E. van den Ban, Invariant differential operators on a semisimple symmetric space and finite multiplicities in a Plancherel formula, Arkiv för Matematik 25 (1987), 175-187.

3. E. van den Ban, The principal series for a reductive symmetric space I, Ann. Sci. École Norm. Sup. 21 (1988), 359-412.

4. Y. Benoist, Multiplicité un pour les éspaces symétriques exponentiel, Mem. Math. Soc. France 15 (1984), 1-37.

5. P. Bernat et al, Représentations des Groupes de Lie Résolubles, Dunod, Paris, 1972.

6. L. Corwin, F. Greenleaf & G. Grélaud, Direct integral decompositions and multiplicities for induced representations of nilpotent Lie groups, Trans. Amer. Math. Soc. 304 (1987), 549-583.

7. M. Duflo, Sur les extensions des répresentations irréductibles des groupes de Lie nilpotents, Ann. Sci. École Norm. Sup. 5 (1972), 71-120.

8. M. Duflo, Théorie de Mackey pour les groupes de Lie algébriques, Acta Math. 149 (1982), 152-213.

9. H. Fujiwara, Représentations monomiales d’un groupe de Lie nilpotente, Pacific J. Math. 127 (1987), 329-352.

10. H. Fujiwara, Représentations monomiales des groupes de Lie résolubles exponentiels, The Orbit Method in Representation Theory, Birkhäuser, 1990, pp. 61-84.

11. R. Lipsman, Harmonic analysis on non-semisimple symmetric spaces, Israel J. Math. 54 (1986), 335-350.

12. R. Lipsman, Orbital parameters for induced and restricted representations, Trans. Amer. Math. Soc. 313 (1989), 433-473.

13. R. Lipsman, Harmonic analysis on exponential solvable homogeneous spaces, Pacific J. Math. 140 (1989), 117-147.

14. R. Lipsman, The Penney-Fujiwara Plancherel formula for symmetric spaces, The Orbit Method in Representation Theory, Birkhäuser, 1990, pp. 135-145.

15. R. Lipsman, Induced representations of completely solvable Lie groups, Annali della Scuola Norm. Pisa 17 (1990), 127-164.

16. R. Lipsman, The Penney-Fujiwara Plancherel formula for abelian symmetric spaces and completely solvable homogeneous spaces, Pacific J. Math. 151 (1991), 265-295.

17. R. Lipsman, The multiplicity function on exponential and completely solvable homogeneous spaces, Geometriae Dedicata 39 (1991), 155-161.

18. R. Lipsman, The Penney-Fujiwara Plancherel formula for homogeneous spaces, Representation Theory of Lie Groups and Lie Algebras, World Scientific Press, 1992, pp. 120-139.

19. R. Lipsman, The Plancherel formula for homogeneous spaces with polynomial spectrum, Pacific J. Math. 159 (1993), 351-377.

20. R. Penney, Abstract Plancherel theorems and a Frobenius reciprocity theorem, J. Functional Analysis 18 (1975), 177-190.

21. R. Penney, Spherical distributions on Lie groups and C^{∞} vectors, Trans. Amer. Math. Soc. 223 (1976), 367-384.

22. L. Pukanszky, Unitary representations of Lie groups with co-compact radical and applications, Trans. Amer. Math. Soc. 236 (1978), 1-49.