THE BEHAVIOR OF FOURIER TRANSFORMS FOR

NILPOTENT LIE GROUPS

Ronald L. Lipsman and Jonathan Rosenberg

Abstract. We study weak analogues of the Paley-Wiener Theorem for both the scalarvalued and the operator-valued Fourier transforms on a nilpotent Lie group G. Such theorems should assert that the appropriate Fourier transform of a function or distribution of compact support on G extends to be “holomorphic” on an appropriate complexification of (a part of) Ĝ. We prove the weak scalar-valued Paley-Wiener Theorem for some nilpotent Lie groups but show that it is false in general. We also prove a weak operator-valued Paley-Wiener Theorem for arbitrary nilpotent Lie groups, which in turn establishes the truth of a conjecture of Moss. Finally, we prove a conjecture about Dixmier-Douady invariants of continuous-trace subquotients of C^{∗}(G) when G is two-step nilpotent.

0. Introduction

The classical Paley-Wiener Theorems characterize the Fourier transforms of various classes of generalized functions of compact support on ℝ^{n}, as classes of holomorphic functions on ℂ^{n} with exponential growth in imaginary directions and satisfying suitable growth or decay conditions in real directions, depending on the class of functions or distributions considered. A long-standing problem in abstract harmonic analysis has been to find analogues of these theorems with ℝ^{n} replaced by a non-commutative locally compact group. In searching for such analogues, one immediately encounters four difficulties: (0.1) It is not clear what the correct analogue of the Fourier transform should be for a

(generalized) function φ on a non-commutative locally compact group G. In general there are two main candidates, the operator-valued Fourier transform φ̂_{op} : π ↦ π(φ), with π ranging over Ĝ, the set of equivalence classes of irreducible unitary representations of G, and (when it makes sense) the scalar-valued Fourier transform φ̂ : π ↦ Tr π(φ). However, depending on the context, it might be appropriate to consider non-unitary representations as well. (Still another candidate for a Fourier transform, based more on the theory of deformation quantization than on representation theory, is discussed in [ArnG]. We will not consider it in this paper.)

(0.2) Since different representations of G act on different Hilbert (or Banach or Fréchet) spaces, and representations are only defined up to equivalence, it is not clear how to study the variation of π(φ) with π.

(0.3) In general, a “dual” object for a non-commutative group, for example Ĝ, has “singularities,” due to the fact that representations tend to come in “series” which

Typeset by AMS-TEX

1991 Mathematics Subject Classification. Primary 22E27; Secondary 43A30, 44A12, 22D25.

Key words and phrases. nilpotent Lie group, scalar-valued Fourier transform, operator-valued Fourier transform, Paley-Wiener Theorem, Radon transform, Dixmier-Douady class.

Both authors were partially supported by NSF grant DMS-92-25063.

can degenerate or run into one another. It is not clear what sort of continuity, let alone holomorphicity, to expect of Fourier transforms at these singularities.

(0.4) When G = ℝ^{n}, Ĝ ≅ ℝ^{n} has a natural complexification, ℂ^{n}. For a general locally compact group G, even leaving aside the problem of singularities, it is not clear how to define an analytic structure on Ĝ.

Our objective in this paper is to discuss some methods for overcoming these difficulties to obtain information about the Fourier transform in the case of nilpotent Lie groups. In fact, connected and simply connected nilpotent Lie groups are the non-commutative groups “closest in appearance” to ℝ^{n}. Although we shall make some progress in studying Fourier analysis on them, we are still very far from a complete understanding.

Let G be a connected and simply connected nilpotent Lie group with Lie algebra 𝔤. The Kirillov orbit method identifies Ĝ with the quotient space 𝔤^{∗}/G of the dual of 𝔤 under the coadjoint action, even as a topological space [Br, Joy]. We denote the irreducible representation attached to an orbit O (well-defined up to unitary equivalence) by π_{O}, or by π_{f} if O = G·f. For φ ∈ C_{c}^{∞}(G) and π ∈ Ĝ, π(φ) is a trace-class operator; furthermore, φ ↦ Tr π(φ) is a tempered distribution on G (though not necessarily a measure, as it is for semisimple Lie groups). Thus for φ ∈ C_{c}^{∞}(G), the scalar-valued Fourier transform of φ is well-defined as a map φ̂ : 𝔤^{∗}/G → ℂ, O ↦ Tr π_{O}(φ). When G is nonabelian, the linear map φ ↦ φ̂ is never injective, but one can hope to characterize its image as a space of functions on 𝔤^{∗}/G satisfying certain continuity and holomorphicity conditions. Such a characterization we will call a strong scalar-valued Paley-Wiener Theorem for G. As far as we know, there is almost nothing in the literature on such theorems, except in very special cases.

The operator-valued Fourier transform φ̂_{op} : φ ↦ π(φ) for a nilpotent Lie group is harder to define as an actual operator-valued function, for the reason discussed in (0.2) above. Nevertheless, the following comments are in order: (0.5) If φ ∈ L^{1}(G), φ̂_{op} makes sense as an element of the C^{∗}-algebra C^{∗}(G) (the universal

C^{∗}-algebra completion of L^{1}(G)). Via the theory of realization of C^{∗}-algebras as algebras of sections of bundles (see e.g. [Fell] or [Ros]), φ̂_{op} can then often be interpreted as a section of a bundle whose fibers are C^{∗}-algebras. If (and only if) this bundle can be trivialized, then the operator-valued Fourier transform of φ can be interpreted as an ordinary operator-valued function. Even when this is the case, the precise form of the function will depend on the trivialization of the bundle.

(0.6) If φ is a tempered distribution on G (and not necessarily an L^{1} function), then for π ∈ Ĝ, π(φ) makes sense as an unbounded closed operator on the Hilbert space of π, or else as a continuous operator on the Fréchet space of C^{∞}-vectors of the the representation. Both interpretations give the same (unambiguous) notion of vanishing of φ̂_{op} at a specific representation π.

(0.7) Since the map φ ↦ φ̂_{op} from L^{1}(G) to C^{∗}(G) is injective, one can hope to describe the operator-valued Fourier transform as a specific bijection from certain spaces of functions or distributions on G to certain subspaces of C^{∗}(G) or spaces of operatorvalued functions. Such a description we will call a strong operator-valued Paley-Wiener Theorem for G. As far as we know, such theorems have been proved only for Heisenberg groups and certain other special cases: see [Ando], [Kum], and [Sev1, Sev2].

There seems to be very little hope for proving either scalar-valued or operator-valued strong Paley-Wiener Theorems for arbitrary nilpotent Lie groups. However, limited progress has been made in proving weaker substitutes. The papers [Moss] and [Park], motivated in part by [ScSi], proposed and managed to prove certain cases of:

0.8. Moss’s Conjecture. Let G be a connected and simply connected nilpotent Lie group, and let φ be a distribution on G with compact support. If φ̂_{op} vanishes on a set of positive Plancherel measure in Ĝ, then φ = 0.

One of the goals of this paper is to prove this conjecture in the generality in which we have stated it. (Moss and Park only studied the case where φ is given by an L^{∞} function, and used methods which do not generalize to arbitrary nilpotent Lie groups.) We will in fact prove in §2 something much more precise (see Corollary 2.15 below), which one can call a weak operator-valued Paley-Wiener Theorem for G. Here we rely on fundamental work of Niels Pedersen [Ped1, Ped2], which asserts that one can partition 𝔤^{∗}/G into a finite union of algebraic varieties defined over ℝ, in such a way that the representations corresponding to the Kirillov orbits in any given set of the partition can all be realized on a fixed Hilbert space in a very convenient way. Once one does this, our weak operator-valued Paley-Wiener Theorem asserts that for φ a distribution with compact support, π ↦ φ̂_{op}(π) is “holomorphic” over each variety in the partition, in the sense that a large collection of its matrix coefficients extend to be analytic functions on the complexification of that variety.

Our search for a proof of Moss’s conjecture led us also to try to prove:

0.9. Conjectured weak scalar-valued Paley-Wiener Theorem. For φ ∈ C_{c}^{∞}(G), the scalar-valued Fourier transform φ̂ extends to be holomorphic on a covering of the complexification of some algebraic variety defined over ℝ which parameterizes the “generic” irreducible unitary representations of G.

Remark. A quick calculation for the Heisenberg group of dimension 2n + 1 shows that this couldn’t be true without passage to a covering, since the formula for φ̂ involves a factor of |ξ|^{−n}, where ξ is the parameter giving the central character of the representation. When n is odd, this factor would appear not to be holomorphic in ξ, but we can rewrite it as (ξ^{2n})^{− \frac{1}{2}} , which now extends to a holomorphic function on the double cover of the punctured ξ-plane. □

In §1 of this paper, we prove this weak scalar-valued Paley-Wiener Theorem for certain nilpotent Lie groups, and show how it leads to a proof of Moss’s conjecture for these groups. More surprisingly, however, we show that Conjecture 0.9 is false in general. The analogue of Moss’s conjecture also fails for the scalar-valued Fourier transform: it is possible for φ̂ to vanish on a set of positive Plancherel measure in Ĝ without vanishing identically. Curiously, this fact is related to the Paley-Wiener Theorem for the classical Radon transform, as discussed in [Hel]. The analysis of certain counterexamples, which is also in §1, shows however that there is a sense in which Conjecture 0.9 is “close enough” to being true so that one can still deduce proofs of some cases of Moss’s conjecture from an analysis of the scalar-valued Fourier transform.

Finally, in §3 of this paper, we study the problem raised in (0.5) above about trivialization of the bundles of C^{∗}-algebras which are implicit in the operator-valued Fourier transform. For 2-step nilpotent groups, we are able to prove a conjecture about this formulated by one of us in [RaRo, §4] and in [Ros, §3], thereby extending results in [Echt, §6].

1. The scalar-valued Fourier transform

In this section we let G denote a connected and simply connected nilpotent Lie group with Lie algebra 𝔤, and we study the scalar-valued Fourier transform φ ↦ φ̂. Note that φ̂ is well-defined as a function on Ĝ if φ ∈ C_{c}^{∞}(G), but that also (by the Plancherel Theorem) φ̂(π) is well-defined for almost all π (with respect to Plancherel measure) if φ = φ_{1} ∗ φ_{2}, φ_{1}, φ_{2} ∈ L^{1}(G) ∩ L^{2}(G). In this latter case, the exceptional null set of π’s where φ̂(π) is undefined will depend on φ.

A weak scalar-valued Paley-Wiener Theorem and Moss’s conjecture for nilpotent Lie groups with square-integrable representations. Conjecture 0.9, the weak scalar-valued Paley-Wiener Theorem, would if true be a natural generalization of the fact that the Fourier transform of φ ∈ C_{c}^{∞}(ℝ^{n}) extends to an entire holomorphic function on ℂ^{n}. In this subsection we will observe that a slightly more precise form of Conjecture 0.9 holds for nilpotent Lie groups with square-integrable representations, and leads immediately to an easy proof of Moss’s Conjecture (0.8) for these groups. Then in the next subsection we will carry out a similar analysis for a much larger class of nilpotent Lie groups, and will find that Conjecture 0.9 fails even for some two-step nilpotent groups.

First we recall some of the theory developed in [MoorWo], for which an alternative reference is [CorGr, §4.5]. Let G be a connected and simply connected nilpotent Lie group. If π is an irreducible unitary representation of G, the projective kernel of π is the set of elements of G mapped under π to scalar operators. This is always a connected normal subgroup N of G, containing the center Z of G. The representation π is said to be square-integrable modulo N if the absolute values of its matrix coefficients descend to L^{2} functions on G/N. By [MoorWo, Theorem 1], this is equivalent to the condition that the Kirillov orbit corresponding to π be flat, i.e., of the form O = f + 𝔫^{⊥}, where 𝔫 is the Lie algebra of N, and also equivalent to the condition that the alternating bilinear form B_{f}, defined by B_{f}(X, Y) = f([X, Y]), should descend to a non-degenerate form on 𝔤/𝔫.

1.1. Theorem. Let G be a connected and simply connected nilpotent Lie group with Lie algebra 𝔤. Let Z and 𝔷 be the centers of G and 𝔤, respectively, and suppose G has irreducible unitary representations which are square-integrable modulo Z. Fix Lebesgue measures on 𝔷 and on 𝔤/𝔷, and take Lebesgue measure on 𝔤 to be the product of these two. Relative to our fixed Lebesgue measure on 𝔤/𝔷, for f ∈ 𝔤^{∗}, we can define P (f), the Pfaffian polynomial of the alternating bilinear form B_{f}, viewed as an alternating form on 𝔤/𝔷. (As a function of f ∈ 𝔤^{∗}, it only depends on the image ḟ of f in 𝔤^{∗}/𝔷^{⊥} = 𝔷^{∗}.) Let V (ℝ) = {ḟ ∈ 𝔷^{∗} : P (f) ≠ 0}, and let V (ℂ) be its complexification {ḟ ∈ 𝔷_{ℂ}^{∗} : P (f) ≠ 0}. Then V is a smooth affine algebraic variety defined over ℝ, connected in the (complex) Zariski topology and with V (ℝ) Zariski-dense in V (ℂ), and there is a natural map ψ : V (ℝ) → 𝔤^{∗}/G having the following properties:

(a) ψ is injective, and the complement in 𝔤^{∗} of the inverse image of ψ(V (ℝ)) has Lebesgue measure zero.

(b) The pull-back under ψ of some quasi-image in 𝔤^{∗}/G of Lebesgue measure on 𝔤^{∗} is quasi-equivalent to a smooth measure on V (ℝ).

(c) For φ ∈ C_{c}^{∞}(G), v ↦ φ̂(ψ(v)) extends to a holomorphic function on a double cover

of V (ℂ). Furthermore, if φ ∈ C_{c}^{∞}(G) and if φ̂_{op} vanishes on a set of positive Plancherel measure in Ĝ, it follows that φ ≡ 0.

Proof. (i) Define ψ by sending ḟ ∈ V (ℝ) to f + 𝔷^{⊥}, where f lies in the inverse image of ḟ in 𝔤^{∗}. By [MoorWo], this gives a bijection from V (ℝ) to the set of G-orbits in 𝔤^{∗} corresponding to irreducible unitary representations which are square-integrable modulo Z. The inverse image of ψ(V (ℝ)) in 𝔤^{∗} is {f ∈ 𝔤^{∗} : P (f) ≠ 0}, which is obviously a Zariskiopen subset of 𝔤^{∗}, and is non-empty by the hypothesis that B_{f} is non-degenerate on 𝔤/𝔷 for at least one f ∈ 𝔤^{∗}. Properties (a) and (b) are obvious. Since V is a non-empty Zariskiopen subset of an affine space, defined by the non-vanishing of the polynomial P which has real coefficients, it is a smooth affine algebraic variety defined over ℝ, connected in the (complex) Zariski topology and with V (ℝ) Zariski-dense in V (ℂ). Furthermore, since V is Zariski-connected, V (ℂ) is connected in the usual Hausdorff topology (for a proof, see [Mum, Corollary 4.16] or [Shaf, Ch. VII, §2]). (Here it is essential to take complex points; V (ℝ) may not be connected in the usual Hausdorff topology.) By [MoorWo, Theorem 6], the Plancherel measure on Ĝ corresponds under ψ to a constant multiple of the smooth measure |P (ḟ)| dḟ on V (ℝ). (The constant only depends on the dimensions of 𝔤 and 𝔷.)

(ii) Now let φ ∈ C_{c}^{∞}(G). We compute the distribution character of π_{ψ(ḟ)} on φ using the inversion theorem for the abelian Fourier transform as in [MoorWo, pp. 458–459], and find that

(1.2)  φ̂(ψ(ḟ)) = Tr π_{ψ(ḟ)}(φ) = \frac{c}{|P (f)|}φ̃(ḟ),

where φ̃ is the (abelian) Fourier transform of (φ ∘ exp)|_{𝔷}, and c is a constant depending only on our normalizations of Lebesgue measures on 𝔤 and 𝔷. Clearly φ̃(ḟ) belongs to the Paley-Wiener class PW(𝔷^{∗}), and so |P (f)|φ̂ ∘ ψ(ḟ) extends to be entire holomorphic on 𝔷_{ℂ}^{∗}. Furthermore, the extra factor |P (f)| may be rewritten as (P (f)^{2})^{\frac{1}{2}} , which makes sense as a holomorphic function on the double cover of V (ℂ), so (c) is satisfied. Finally, if φ̂ ∘ ψ vanishes on a set of positive smooth measure in V (ℝ), we deduce that its analytic continuation on the double cover of V (ℂ) vanishes on a set of positive smooth measure, and thus must vanish everywhere since the double cover of V (ℂ) is connected.

(iii) Now suppose φ̂_{op}(π) = π(φ) vanishes for π in a set E of positive Plancherel measure in Ĝ. Let λ be the left regular representation of G. Then for any g ∈ G, π(λ(g)φ) = π(g)π(φ) also vanishes for π ∈ E, and in particular its trace is zero. So { \atop λ(g)φ} vanishes on E, and by (ii) above, { \atop λ(g)φ} ∘ ψ ≡ 0. Now by the Plancherel Theorem, we have

φ(g^{−1}) = (λ(g)φ)(1) = const ∫_{V (ℝ)} { \atop λ(g)φ} ∘ ψ(ḟ) |P (ḟ)| dḟ = 0

for all g, i.e., φ ≡ 0. □

1.3. Corollary. If G has square-integrable representations modulo Z, then Moss’s Conjecture (0.8) holds for arbitrary distributions φ of compact support on G.

Proof. It is sufficient to observe that Moss’s Conjecture for functions in C_{c}^{∞}(G) implies the same conjecture for arbitrary distributions φ of compact support on G. Suppose φ is such a distribution and φ̂_{op} vanishes on a set E of positive Plancherel measure. Then for any function θ ∈ C_{c}^{∞}(G), { \atop φ ∗ θ}_{op} = φ̂_{op} · θ̂_{op} also vanishes on E. Since the convolution φ ∗ θ is in C_{c}^{∞}(G), we deduce from Theorem 1.1 that φ ∗ θ = 0. This being true for all θ, φ = 0 as a distribution. □

Failure of the weak scalar-valued Paley-Wiener Theorem, but a proof of Moss’s conjecture, for groups whose generic orbits are flat. We proceed now to examine what happens to the proof of Theorem 1.1 when G is a connected, simply connected nilpotent Lie group whose generic orbits are flat, i.e., whose generic irreducible unitary representations are square-integrable modulo their projective kernels, but for which the projective kernel varies depending on the representation. As we shall see, the class of groups with this property includes all two-step nilpotent Lie groups, and some higher-step groups as well.

1.4. Theorem. Let G be a connected and simply connected nilpotent Lie group with Lie algebra 𝔤, and let 2d be the maximal dimension of G-orbits in 𝔤^{∗}. Assume that there is a non-empty G-invariant Zariski-open subset Ω of 𝔤^{∗} consisting of flat orbits of dimension 2d. Then (after perhaps modifying Ω, keeping the same conditions), there is a smooth quasi-projective variety W (ℝ) parametrizing a set of abelian ideals 𝔫◁𝔤 of codimension 2d, such that Ω/G can be identified with a Zariski-open subset V (ℝ) of

E(ℝ) = {(𝔫, f) : 𝔫 ∈ W (ℝ), f ∈ 𝔫^{∗}, f ⊥ [𝔤, 𝔫]}.

Here V (ℝ) = Ω/G and E(ℝ) are each the set of real points of a quasi-projective algebraic variety defined over ℝ, and Φ : E(ℝ) → W (ℝ) sending (𝔫, f) to 𝔫 is an algebraic vector bundle. The Plancherel measure class on Ĝ corresponds to the smooth measure class on V (ℝ). For φ ∈ C_{c}^{∞}(G), the restriction of φ̂ to any fiber of Φ extends to be holomorphic on a double cover of the complexification of that fiber. If φ̂_{op} vanishes on a set of positive Plancherel measure in Ĝ, then φ = 0. However, it is possible for φ̂ to be non-zero on a set of positive Plancherel measure and yet to vanish identically on another set of positive Plancherel measure.

Proof. Let U(ℝ) be the Grassmann manifold of 2d-dimensional linear subspaces of 𝔤^{∗}, which by duality we can identify with the Grassmann manifold of 2d-codimensional linear subspaces of 𝔤. As is well known, this can be identified with the real points of a smooth projective variety U. The “Gauss map” Φ : Ω → U(ℝ) sending f ∈ Ω to the tangent space at f to the variety G · f is a morphism of smooth varieties (since all the orbits in Ω have the same dimension 2d), and Φ is constant on G-orbits by the assumption that the orbits in Ω are flat. The image of Φ is contained in the set Y of annihilators of abelian ideals^{1} 𝔫 in 𝔤 of codimension 2d, and this is a closed algebraic subset of U(ℝ) since being an abelian ideal is equivalent to a finite number of (real) polynomial conditions. Cutting Ω down if necessary to a Zariski-open subset (see for example [Shaf, Ch. I, §5, Theorem 6]), we may suppose the image of Φ is a smooth variety W (ℝ), which will consist of annihilators of abelian ideals 𝔫 in 𝔤 of codimension 2d which are also of the form 𝔤_{f} for some f ∈ 𝔤^{∗}. For any such 𝔫, one has square-integrable representations modulo 𝔫 corresponding to all f ∈ 𝔫^{∗} ∩ [𝔤, 𝔫]^{⊥} for which the Pfaffian P_{𝔫}(f) of the induced form on 𝔤/𝔫 is non-zero. For any such pair (𝔫, f), let φ ∈ C_{c}^{∞}(G) and O = f + 𝔫^{⊥}. We compute the distribution character of π_{O} on φ using the inversion theorem for the abelian Fourier transform as in [MoorWo, pp. 458–459], and find that

(1.5)  φ̂(O) = Tr π_{O}(φ) = \frac{c}{|P_{𝔫}(f)|}φ̃(f),

1They are ideals since the orbits are flat [CorGr, Theorem 3.2.3], and are abelian by the condition that the orbits have maximal dimension [Dix2, Proposition 1.11.7].

where φ̃ is the (abelian) Fourier transform of (φ ∘ exp)|_{𝔫}, and c is a constant depending only on the normalizations of Lebesgue measures on 𝔤 and 𝔫. (Note here that for fixed O = f + 𝔫^{⊥}, f is only defined up to an element of 𝔫^{⊥}, but gives a well-defined class in 𝔤^{∗}/𝔫^{⊥} = 𝔫^{∗}, so that φ̃(f) is well-defined. Similarly, P_{𝔫}(f) only depends on the image of f in 𝔫^{∗}.) Since the polynomial P_{𝔫} is not identically zero, there is a non-empty Zariski-open subset of (𝔫/[𝔤, 𝔫])^{∗} on which P_{𝔫}(f) ≠ 0, and the character formula (1.5) is valid for all such f. Clearly φ̃(f) belongs to the Paley-Wiener class PW(𝔫^{∗}), and so |P_{𝔫}(f)|φ̂(G · f) extends to be entire holomorphic on 𝔫_{ℂ}^{∗}. By general Kirillov theory, Plancherel measure on Ĝ must correspond to a smooth measure on V (ℝ). From the character formula (1.5), if φ ∘ exp vanishes identically on a set of 𝔫’s of positive measure in W (ℝ), then φ̂ will vanish on a set of positive Plancherel measure. This can happen without φ vanishing identically. (A specific counterexample will be given in Example 1.8 below.)

Now let us verify Moss’s conjecture for G. By the proof of Corollary 1.3, it is enough to verify the conjecture for φ ∈ C_{c}^{∞}(G). Assume that for such a φ, φ̂_{op} vanishes on a set E of positive Plancherel measure. By Fubini’s Theorem, there is a set E_{1} of positive smooth measure in W (ℝ) such that for each 𝔫 ∈ E_{1}, φ̂_{op} vanishes on a set of π_{O}’s with O = f +𝔫^{⊥} and with f ranging over a set of positive Lebesgue measure in (𝔫/[𝔤, 𝔫])^{∗}. Then for any g ∈ G and 𝔫 ∈ E_{1}, { \atop λ(g)φ} vanishes at f + 𝔫^{⊥} for f ranging over a set of positive Lebesgue measure in (𝔫/[𝔤, 𝔫])^{∗}. Since φ̃(f) is holomorphic on 𝔫_{ℂ}^{∗}, (1.5) implies that the abelian Fourier transform of (λ(g)φ) ∘ exp |_{𝔫} vanishes identically on (𝔫/[𝔤, 𝔫])^{∗} for all g ∈ G and all 𝔫 ∈ E_{1}, and thus for such 𝔫 and any g, the integral of (λ(g)φ) ∘ exp over any coset in 𝔫 of [𝔤, 𝔫] vanishes by Fourier inversion. Since

∫_{[𝔤, 𝔫]}(λ(exp −Y)φ) ∘ exp(X)dX = ∫_{[𝔤, 𝔫]} φ ∘ exp(Y + X)dX,

(the Jacobian of the change of variables that comes in is 1 since 𝔤 is nilpotent), the integral of φ∘exp over any coset in 𝔤 of [𝔤, 𝔫] vanishes. Let r = dim[𝔤, 𝔫] = dim W (ℝ), and consider the r-dimensional Radon transform

{ \atop φ ∘ exp}_{Radon}(σ, X) = ∫_{σ} φ ∘ exp(Y + X) dY,

where σ ranges over r-dimensional subspaces of 𝔤. We have shown that { \atop φ ∘ exp}_{Radon}(σ, X) vanishes identically for σ = [𝔤, 𝔫], 𝔫 ∈ E_{1}. For f ∈ 𝔤^{∗}, k ∈ ℕ, define the polynomial

P_{k}(f) = ∫_{𝔤} φ ∘ exp(X)f(X)^{k} dX.

If this vanishes for all k, then φ ≡ 0 (using Stone-Weierstrass, for example). By the Paley-Wiener Theorem for the r-dimensional Radon transform [Hel, Ch. I, Corollary 6.4], what we know is that the restriction of P_{k} to [𝔤, 𝔫]^{⊥} vanishes for each 𝔫 ∈ E_{1}. This is enough to imply that each P_{k} vanishes and thus that φ vanishes. □

1.6. Proposition. The hypotheses of Theorem 1.4 are satisfied if G is a two-step connected and simply connected nilpotent Lie group. In fact, for such a group, any coadjoint orbit is flat.

Proof. Let 𝔤 be a two-step nilpotent Lie algebra with center 𝔷, and let the corresponding simply connected nilpotent Lie groups be G and Z. The two-step assumption means [𝔤, 𝔤] ⊆ 𝔷. If f ∈ 𝔤^{∗} and X ∈ 𝔤, then

(exp(tX) · f) (Y) = f (e^{−t ad X}Y )
= f (Y − t[X, Y] + \frac{t^{2}}{2} [X, [X, Y]] + ⋯)
= f (Y − t[X, Y]) = (f − tf ∘ ad X) (Y),

so G · f = f + 𝔤_{f}^{⊥} and the G-orbit through f is flat, with G_{f} as the projective kernel of the corresponding irreducible unitary representation. So the Zariski-open subset consisting of orbits of maximal dimension (see [Dix2, Proposition 1.11.5]) will satisfy the hypotheses of Theorem 1.4. □

1.7. Remark. However, as observed in [MoorWo, p. 453], there is no upper bound on the nilpotent length of a nilpotent Lie group with generic representations square-integrable modulo the center, hence a fortiori modulo their projective kernels. So Theorem 1.4 also applies to some higher-step groups.

1.8. Example. Here is a specific example of the situation in Theorem 1.4. Let 𝔤 be the free two-step nilpotent Lie algebra on 3 generators, with basis X_{1}, …, X_{6}, where [X_{6}, X_{5}] = X_{3}, [X_{6}, X_{4}] = X_{1}, [X_{5}, X_{4}] = X_{2}, and X_{1}, X_{2}, X_{3} are central. (In [Ped3], this example is labelled N6N15.) Let G be the corresponding group. The center 𝔷 of 𝔤 is spanned by X_{1}, X_{2}, X_{3}. If f ∈ 𝔤^{∗} is “regular,” then 𝔤_{f} is an ideal containing 𝔷 with codimension 1, and any such ideal arises in this way. Thus we see that 𝔤_{reg}^{∗}/G can be identified with a Zariski-open subset of the total space of a rank-two vector bundle over the set of possibilities for 𝔫 = 𝔤_{f}, which in turn can be identified with the projective plane of one-dimensional subspaces of 𝔤/𝔷. For any such 𝔫, [𝔤, 𝔫] is a two-dimensional subspace of 𝔷, and 𝔫 and [𝔤, 𝔫] determine one another. In this case P_{𝔫} is a linear functional on (𝔫/[𝔤, 𝔫])^{∗}. From (1.5), φ̂(f + 𝔫^{⊥})|P_{𝔫}(f)| is (up to a constant) the abelian Fourier transform of φ ∘ exp |_{𝔫}. Clearly one can choose φ ≠ 0 ∈ C_{c}^{∞}(G) so that the support of φ ∘ exp does meet some possible 𝔫 but misses a set of 𝔫’s of positive measure in the projective plane of one-dimensional subspaces of 𝔤/𝔷. So vanishing of φ̂ on a set of positive Plancherel measure does not imply vanishing of φ̂ everywhere.

In fact,

∫_{(𝔫/[𝔤, 𝔫])^{∗}} φ̂(f + 𝔫^{⊥})|P_{𝔫}(f)| dḟ = c ∫_{[𝔤, 𝔫]} φ ∘ exp .

So we can compute in this way the integral of φ ∘ exp over any two-dimensional subspace of 𝔷, in other words the Radon transform of φ ∘ exp |_{𝔷}, restricted only to those two-planes in 𝔷 which pass through the origin. Now we can relate the Plancherel formula for G to the classical Radon inversion formula [Hel, Ch. I], which says that

φ(1) = \frac{−1}{2π} ∫\begin{matrix}ω∈𝔷 \\ |ω|=1 \end{matrix} \frac{d^{2}}{dp^{2}} \{∫\begin{matrix}z∈𝔷 \\ z·ω=p \end{matrix} φ ∘ exp\}_{p=0} dω.

Rewriting this in terms of φ̂ gives precisely the Plancherel formula for G. And a correct formulation of a weak Paley-Wiener Theorem for the scalar-valued Fourier transform on G resembles the Paley-Wiener Theorem for the Radon transform in [Hel, Ch. I].

1.9. Remark. It should be clear from Theorem 1.4 and Example 1.8 that the weak Paley-Wiener Theorem for the scalar-valued Fourier transform can fail whenever the supports of the distribution characters of “generic” representations of G “rotate around.” In the examples we’ve examined here, the support of the distribution character has always been an ideal in 𝔤. While this is the case for low-dimensional nilpotent Lie groups, Pukanszky [Puk, p. 413] has given an interesting 7-dimensional example where the supports of the distribution characters of “generic” representations are quadric cones in 𝔤. The weak Paley-Wiener Theorem for the scalar-valued Fourier transform definitely fails in such cases as well.

2. A Paley-Wiener Theorem for the operator-valued Fourier transform

Pedersen’s models for the irreducible unitary representations. In an important series of papers ([Ped1], [Ped2], which built on ideas of Pukanszky), Pedersen has given a detailed description of the fine structure of Ĝ, for a nilpotent Lie group G, and he has elucidated very sharply the nature of the analytic objects that appear in that structure. In this section we utilize these objects to prove a weak version of the operator-valued Paley-Wiener Theorem.

The structure depends on a choice of a Jordan-Hölder basis for 𝔤 which we now fix: namely, a basis X_{1}, X_{2}, …, X_{m} for 𝔤 with the property that 𝔤_{j} = span(X_{1}, …, X_{j}) is an ideal in 𝔤 for each 1 ≤ j ≤ m. Each f ∈ 𝔤^{∗} defines a bilinear form B_{f} by B_{f}(X, Y) = f([X, Y]). The restriction of B_{f} to each 𝔤_{j} determines a set of jump indices

(2.1)  J_{f}^{j} = {1 ≤ k ≤ j : X_{k} ∉ 𝔤_{k−1} + rad (B_{f}|_{𝔤_{j}})} ,

and 𝔤^{∗} is the finite union of the sets Ω_{ε} where the m-tuple of sets of jump indices (J_{f}^{1}, …, J_{f}^{m}) takes a fixed value ε. This is Pedersen’s (“fine”) stratification of 𝔤^{∗}. Each Ω_{ε} is G-invariant. The non-empty sets Ω_{ε}, of which there are only a finite number, may be numbered as Ω_{1}, …, Ω_{k} so that Ω_{1} is Zariski-open in 𝔤^{∗} and each successive Ω_{j} is a Zariski-open set in 𝔤^{∗}\(⋃_{i<j} Ω_{i}), hence is a real affine variety. Note that the last set in the stratification, Ω_{k}, corresponds to the case where ε = ∅, and thus to the one-dimensional unitary representations of G. For any particular value of ε, all the orbits in Ω_{ε} have a fixed dimension 2d, which is also the cardinality of J_{f}^{m}, say J_{f}^{m} = {j_{1} < ⋯ < j_{2d}}. Then X_{1}, X_{2}, …, X_{m} can be viewed as linear coordinates ξ_{1}, ξ_{2}, …, ξ_{m} on 𝔤^{∗}, and by [Ped2, Lemma 1.6.1], each G-orbit in Ω_{ε} contains a unique point where ξ_{j_{1}}, …, ξ_{j_{2d}} each take the value 0. Thus the intersection Λ_{ε} of Ω_{ε} with the hyperplane

{ξ_{j_{1}} = 0, …, ξ_{j_{2d}} = 0}

is an affine variety parametrizing the G-orbits in Ω_{ε}. The first basic result that we need is one of the main results of [Ped2]:

2.2. Theorem ([Ped2, Thm. 2.7.2]—see also [Ped2, Thm. 3.2.1]). Let O be an orbit in 𝔤^{∗} of dimension 2d. Then there exists a unique irreducible unitary representation π of G on L^{2}(ℝ^{d}) such that H_{π}^{∞} = S(ℝ^{d}) and such that for every X ∈ 𝔤, there exist d + 1 polynomial functions a_{X,r} (0 ≤ r ≤ d), with real coefficients, in d variables “associated to O,” which satisfy

(2.3)  dπ(X)f(t) = ia_{X,0}(t)f(t) + ∑_{r=1}^{d} a_{X,r}(t)\frac{∂f}{∂t_{r}} (t) + \frac{1}{2} ∑_{r=1}^{d} \frac{∂a_{X,r}}{∂t_{r}} (t)f(t),

for all f ∈ S(ℝ^{d}). The representation π is in the equivalence class of irreducible unitary representations associated with the orbit O.

What is derived in [Ped2], but not stated explicitly there, is the dependence of the polynomial functions a_{X,r} on the affine parameters that determine O (or π). That computation constitutes the heart of [Ped2, §5]. We summarize the results of Pedersen’s computation in a second theorem.

2.4. Theorem. Let Ω_{ε} be any stratum in Pedersen’s stratification, and let 2d be the dimension of orbits in this stratum. Parameterize the orbits in this stratum by the crosssection Λ_{ε}. Fix X ∈ 𝔤. Then the functions a_{X,r}^{λ} of Theorem 2.1 (0 ≤ r ≤ d), viewed as also depending on λ ∈ Λ_{ε}, are polynomials in t with coefficients that are rational functions of λ. The denominators of these rational functions do not vanish on the complexification (Λ_{ε})_{ℂ} of Λ_{ε}.

Proof. See [Ped2, §5] and [Ped4, Remark 1.4.5]. It is shown in fact that the denominators of these rational functions may be expressed as products of certain explicit Pfaffians, nonvanishing on (Λ_{ε})_{ℂ}. □

Two representative examples. Now we give two examples taken from [Ped3], which will show how Theorems 2.2 and 2.4 can be applied to prove holomorphicity of the matrix coefficients in the affine variables that parameterize the orbits in a single stratum. While these examples are fairly straightforward, they illustrate virtually all of the features that show up in the general case.

2.5. Example. Let 𝔤 be the four-dimensional Lie algebra with Jordan-Hölder basis X_{1}, X_{2}, X_{3}, X_{4}, where [X_{4}, X_{3}] = X_{2} and [X_{4}, X_{2}] = X_{1}, denoted by N4N1 in [Ped3]. Then X_{1}, X_{2}, X_{3}, X_{4} can be considered as linear coordinates ξ_{1}, ξ_{2}, ξ_{3}, ξ_{4} on 𝔤^{∗}, and the open stratum Ω_{1} is given by ξ_{1} ≠ 0. The corresponding jump indices are j_{1} = 2, j_{2} = 4. A cross-section for the orbits is given by Λ_{1}, the intersection of Ω_{1} with the hyperplane {ξ_{2} = ξ_{4} = 0}. Thus coordinates on Λ_{1} can be taken to be ξ_{1} ≠ 0 and ξ_{3}. In this case d = 1, and (2.3) becomes

dπ(X_{j}) = \{ \begin{matrix}iξ_{1}, & j = 1, \\ it_{1}, & j = 2, \\ \frac{1}{2ξ_{1}} (2iξ_{1}ξ_{3} − iξ_{2}^{2} + it_{1}^{2}), & j = 3, \\ ξ_{1} \frac{∂}{∂t_{1}} , & j = 4, \end{matrix}

all acting on S(ℝ). (Here t_{1} is the coordinate on ℝ.)

Integrating, we see that

π(exp x_{j}X_{j}) = \{ \begin{matrix}e^{ix_{1}ξ_{1}}, & j = 1, \\ e^{ix_{2}t_{1}}, & j = 2, \\ exp (\frac{ix_{3}}{2ξ_{1}} (2ξ_{1}ξ_{3} − ξ_{2}^{2} + t_{1}^{2})) , & j = 3, \\ translation by x_{4}ξ_{1}, & j = 4, \end{matrix}

all acting on L^{2}(ℝ).

Now let f_{1}, f_{2} ∈ S(ℝ). We can compute the matrix coefficient ⟨π(g)f_{1}, f_{2}⟩ for fixed g ∈ G as a function of ξ_{1} and ξ_{3}, after setting ξ_{2} = ξ_{4} = 0:

⟨π(exp x_{1}X_{1})π(exp x_{2}X_{2})π(exp x_{3}X_{3})π(exp x_{4}X_{4})f_{1}, f_{2}⟩

= ∫_{ℝ} exp (i (x_{1}ξ_{1} + x_{2}t + x_{3}ξ_{3} + \frac{x_{3}t^{2}}{2ξ_{1}} )) f_{1}(t + x_{4}ξ_{1})\overline{f_{2}(t)} dt

= e^{iξ_{1}x_{1}+iξ_{3}x_{3}} ∫_{ℝ} e^{i(x_{2}t+ \frac{x_{3}}{2ξ_{1}} t^{2})}f_{1}(t + x_{4}ξ_{1})\overline{f_{2}(t)} dt.

The exponential factors and the factor involving f_{1} are clearly everywhere holomorphic in ξ_{1} and ξ_{3} away from {ξ_{1} = 0}, provided that f_{1} has (ordinary) Fourier transform f̂_{1} ∈ C_{c}^{∞}(ℝ) and thus f_{1} lies in the Paley-Wiener class PW(ℝ) (so that f_{1}(t + x_{4}ξ_{1}) is entire in ξ_{1}). If f_{2} ∈ C_{c}^{∞}(ℝ), there is no problem with convergence of the integral, so the matrix coefficient is holomorphic in (Λ_{1})_{ℂ}.

Caution. Because of the factor f_{1}(t + x_{4}ξ_{1}) in the integrand and the growth of the factor exp (i\frac{x_{3}t^{2}}{2ξ_{1}} ) in t for imaginary ξ_{1}, the matrix coefficient will not be holomorphic in ξ_{1} for arbitrary f_{1}, f_{2} ∈ S(ℝ). Indeed, if x_{1} = x_{2} = x_{3} = 0, the matrix coefficient reduces to

⟨π(exp x_{4}X_{4})f_{1}, f_{2}⟩ = g(x_{4}ξ_{1}),

where g = f_{1} ∗ f_{2}^{∗}, f_{2}^{∗}(t) = \overline{f_{2}(−t)}. This is a Schwartz function but it may or may not be real analytic, let alone entire.

2.6. Example. Let 𝔤 be the five-dimensional Lie algebra with Jordan-Hölder basis X_{1}, X_{2}, X_{3}, X_{4}, X_{5}, where [X_{5}, X_{4}] = X_{3}, [X_{5}, X_{3}] = X_{2}, [X_{5}, X_{2}] = X_{1} and [X_{4}, X_{3}] = X_{1}, denoted by N5N6 in [Ped3]. Then, as in the preceding example, X_{1}, X_{2}, X_{3}, X_{4}, X_{5} can be considered as linear coordinates ξ_{1}, ξ_{2}, ξ_{3}, ξ_{4}, ξ_{5} on 𝔤^{∗}, and the open stratum Ω_{1} is given by ξ_{1} ≠ 0. The corresponding jump indices are j_{1} = 2, j_{2} = 3, j_{3} = 4 and j_{4} = 5. A cross-section for the orbits is given by Λ_{1}, the intersection of Ω_{1} with the hyperplane {ξ_{2} = ξ_{3} = ξ_{4} = ξ_{5} = 0}. In this case d = 2, and (2.3) becomes

dπ(X_{j}) = \{ \begin{matrix}iξ_{1}, & j = 1, \\ it_{1}, & j = 2, \\ it_{2}, & j = 3, \\ ξ_{1} \frac{∂}{∂t_{2}} , & j = 4 \\ \frac{1}{2ξ_{1}} (2ξ_{1}^{2} \frac{∂}{∂t_{1}} + 2ξ_{1}t_{1} \frac{∂}{∂t_{2}} − it_{2}^{2}), & j = 5, \end{matrix}

all acting on S(ℝ^{2}).

Integrating, we see that

π(exp x_{j}X_{j})f(t_{1}, t_{2}) = \{ \begin{matrix}e^{ix_{1}ξ_{1}}f(t_{1}, t_{2}), & j = 1, \\ e^{ix_{2}t_{1}}f(t_{1}, t_{2}), & j = 2, \\ e^{ix_{3}t_{2}}f(t_{1}, t_{2}), & j = 3, \\ f(t_{1}, t_{2} + x_{4}ξ_{1}), & j = 4, \\ e^{i(∗)}f(t_{1} + x_{5}ξ_{1}, t_{2} + x_{5}t_{1} + \frac{1}{2}x_{5}^{2}ξ_{1}), & j = 5, \end{matrix}

all acting on L^{2}(ℝ^{2}). Here the factor in the exponential in the formula for

π(exp x_{5}X_{5})f(t_{1}, t_{2})

is quite complicated:

(∗) = −\frac{x_{5}}{2ξ_{1}} t_{2}^{2} − \frac{x_{5}^{2}}{2ξ_{1}} t_{1}t_{2} − \frac{x_{5}^{3}}{6} (t_{2} + \frac{t_{1}^{2}}{ξ_{1}} ) − \frac{x_{5}^{4}}{8} t_{1} − \frac{x_{5}^{5}}{40} .

Now let f_{1}, f_{2} ∈ S(ℝ^{2}). The matrix coefficient ⟨π(g)f_{1}, f_{2}⟩, for fixed g ∈ G, as a function of ξ_{1} (when ξ_{2} = ξ_{3} = ξ_{4} = ξ_{5} = 0), is:

⟨π(exp x_{1}X_{1})π(exp x_{2}X_{2})π(exp x_{3}X_{3})π(exp x_{4}X_{4})π(exp x_{5}X_{5})f_{1}, f_{2}⟩
= ∫_{ℝ} exp (i (x_{1}ξ_{1} + x_{2}t_{1} + x_{3}t_{2} + (∗)))
f_{1}(t_{1} + x_{5}ξ_{1}, t_{2} + x_{4}ξ_{1} + x_{5}t_{1} + \frac{1}{2}x_{5}^{2}ξ_{1})\overline{f_{2}(t_{1}, t_{2})} dt_{1}dt_{2}

The exponential factor multiplied by the factor involving f_{1} is clearly everywhere holomorphic in ξ_{1} away from {ξ_{1} = 0}, provided that f_{1} has (ordinary) Fourier transform f̂_{1} ∈ C_{c}^{∞}(ℝ^{2}) and thus f_{1} lies in the Paley-Wiener class PW(ℝ^{2}). Again, integration against the f_{2} factor presents no difficulty provided f_{2} ∈ C_{c}^{∞}(ℝ^{2}). So for f_{1} ∈ PW(ℝ^{2}), f_{2} ∈ C_{c}^{∞}(ℝ^{2}), the matrix coefficient is holomorphic in {ξ_{1} ≠ 0}.

Holomorphicity of matrix coefficients in the orbital parameters. We shall now deduce, as a consequence of Theorems 2.2 and 2.4, and a lemma which codifies the behavior exhibited by the two examples in the last subsection, the holomorphicity of the matrix coefficients of the irreducible unitary representations as a function of the orbital parameters. First, we show that in fact the matrix coefficients coming from the elements of the enveloping algebra are holomorphic. For the proof of the main result we only need that assertion for the elements of the Lie algebra, but we think the more general fact is interesting enough to warrant explicit mention.

2.7. Proposition. For any element E ∈ U(𝔤) (the universal enveloping algebra of 𝔤) and for every pair f_{1}, f_{2} ∈ S(ℝ^{d}), the function

⟨dπ_{λ}(E)f_{1}, f_{2}⟩

extends to a (rational) holomorphic function of λ defined in the complexification (Λ_{ε})_{ℂ} of Λ_{ε}.

Proof. We apply Theorem 2.4 to the basis vectors {X_{j}}. Then it is clear that on the complexification (Λ_{ε})_{ℂ} of the variety Λ_{ε}, any infinitesimal matrix coefficient

⟨dπ_{λ}(X_{j})f_{1}, f_{2}⟩

will be holomorphic (and rational) in λ. Since the matrix coefficient for any X ∈ 𝔤 will be a finite linear combination of these, the same is true of those matrix coefficients. Finally, from the form of the operators in (2.3), we see that the operators dπ_{λ}(E), E ∈ U(𝔤), involve only further products and derivatives of the polynomial functions a_{X,r}; and hence the matrix coefficients ⟨dπ_{λ}(E)f_{1}, f_{2}⟩ are also rational with no poles in (Λ_{ε})_{ℂ}. □

In order to “integrate” the last result to obtain the main result, we need to compute the one-parameter unitary group of operators corresponding to the operator (2.3). This requires a further examination of Pedersen’s coefficient functions a_{X,r}^{λ}, but this time we focus more carefully on their dependence on the parameter t.

2.8. Lemma. Fix a stratum Ω_{ε} and the corresponding affine cross-section Λ_{ε}. Let 2d be the dimension of orbits in this stratum. Then there is an ordering of the variables t_{1}, …, t_{d} so that the coefficient functions a_{X_{j},r}^{λ}, 1 ≤ r ≤ d, viewed as polynomials in t = (t_{1}, …, t_{d}) ∈ ℝ^{d}, have the following property: For any j, 1 ≤ j ≤ m, the operator dπ(X_{j}) is of the form

(2.9)  dπ(X_{j})f(t) = ia_{X_{j},0}^{λ}(t)f(t) + ∑_{r=1}^{d} a_{X_{j},r}^{λ}(t)\frac{∂f}{∂t_{r}} (t),

where a_{X_{j},r}^{λ}(t) depends only on t_{i}, i < r, 1 ≤ r ≤ d. (In particular, the terms \frac{∂a_{X_{j} ,r}^{λ}}{∂t_{r}} (t)f(t) in (2.3) vanish.)

Proof. Pedersen obtains the formula (2.3) by showing that one can write

ψ^{X_{j}} = a_{X_{j},0}^{λ}(q) + ∑_{r=1}^{d} a_{X_{j},r}^{λ}(q)p_{r},

where ψ^{X_{j}} is the restriction to the orbit O corresponding to λ of the linear functional X_{j} on 𝔤^{∗}, and where the p’s and q’s are canonical symplectic coordinates on O which he constructs. Now recall that since the X_{j}’s are a Jordan-Hölder basis for the nilpotent Lie algebra 𝔤, we have [X_{j}, X_{s}] ∈ 𝔤_{s−1} for any j and s. In terms of symplectic geometry, this means that

{ψ^{X_{j}}, ψ^{X_{s}}} ∈ span (ψ^{X_{1}}, …, ψ^{X_{s−1}}) ,

where { , } is the Poisson bracket on O. Now we need to carefully examine the arguments in [Ped2, §5]. Pedersen divides the jump indices of (2.1) into two disjoint ordered subsets, each containing d elements: the “low” jump indices l̄_{1} < ⋯ < l̄_{d} and the “high” jump indices k_{1} < ⋯ < k_{d}. For simplicity of notation we drop the bars on the l̄_{j}’s and also drop the implicit λ’s everywhere. Then Pedersen shows that ψ^{X_{j}} is a scalar for j < l_{1}, while ψ^{X_{l_{1}}} = q_{1}. Thus, using the facts that the q’s Poisson-commute and that {p_{r}, q_{1}} = δ_{r1}, we have for any j:

{ψ^{X_{j}}, ψ^{X_{l_{1}}} } = {a_{X_{j},0}(q) + ∑_{r=1}^{d} a_{X_{j},r}(q)p_{r}, q_{1}}
= a_{X_{j},1}(q) ∈ span (ψ^{X_{1}}, …, ψ^{X_{l_{1}−1}})
= scalar,

which is the case of the Lemma for r = 1. To get the other cases, observe from Pedersen’s inductive construction of the q_{r}’s that

ψ^{X_{lr}} = q_{r} + polynomial(q_{1}, …, q_{r−1}),

and also that this marks the first occurrence of q_{r}. So one obtains similarly

{ψ^{X_{j}}, ψ^{X_{lr}} } = {a_{X_{j},0}(q) + ∑_{s=1}^{d} a_{X_{j},s}(q)p_{s}, q_{r} + polynomial(q_{1}, …, q_{r−1})}
= a_{X_{j},r}(q) + polynomial(q_{1}, …, q_{r−1})
= polynomial(q_{1}, …, q_{r−1}),

which gives the general case of the Lemma. □

2.10. Remark. It is evident, for example from [Ped4], that Lemma 2.8 could not be true for a completely solvable group. It is the nilpotence that causes the latter terms in (2.3) to disappear. Now the simpler form (2.9) of the infinitesimal operators of the representation allows us to integrate them.

2.11. Proposition. Consider a differential operator D on S(ℝ^{d}) ⊆ L^{2}(ℝ^{d}) of the form

Df(t) = ip(t)f(t) + ∑_{r=1}^{d} q_{r}(t)\frac{∂f}{∂t_{r}} (t),

where p and the q_{r}, 1 ≤ r ≤ d, are real polynomial functions, and q_{r}(t) depends only on t_{j}, j < r. Then for any s ∈ ℝ,

e^{sD}f(t) = e^{ih(s, t)}f(t + k(s, t)),

where h and k are polynomials whose coefficients are explicit polynomials in the coefficients of p and the q_{r}, 1 ≤ r ≤ d. The polynomial k_{r}(s, t) satisfies k_{r}(0, t) = 0, and only involves those t_{j}’s with j < r.

Proof. Let f ∈ S(ℝ^{d}) and set g(s, t) = e^{sD}f(t). Then g satisfies the Cauchy problem

(2.12)  \{ \begin{matrix}g(0, t) = f(t), \\ \frac{∂g}{∂s} = Dg(s, t). \end{matrix}

Since the solution of this problem is unique, it is sufficient to verify that there is a solution of (2.12) of the form

g(s, t) = e^{ih(s, t)}f(t + k(s, t)),

with k_{r}(0, t) = 0 for all r, and k_{r}(s, t) only involving those t_{j}’s with j < r. Substituting the expression for g in (2.12), we see that if there is such a solution, we need to take h(0, t) = 0. Furthermore, equating coefficients of e^{ih}f and of e^{ih} \frac{∂f}{∂t_{r}} in (2.12) gives the partial differential equations

\{ \begin{matrix}\frac{∂h}{∂s} & = p(t) + ∑_{r} q_{r}(t)\frac{∂h}{∂t_{r}} , \\ \frac{∂k_{r}}{∂s} & = q_{r}(t) + ∑_{j<r} q_{j}(t)\frac{∂k_{r}}{∂t_{j}} . \end{matrix}

These equations can be solved by a recursion technique. Write

\{ \begin{matrix}h(s, t) & = sp(t) + s^{2}h_{2}(t) + s^{3}h_{3}(t) + ⋯, \\ k_{r}(s, t) & = sq_{r}(t) + s^{2}k_{2}^{r}(t) + s^{3}k_{3}^{r}(t) + · · · . \end{matrix}

Substituting these expansions into the partial differential equations, one can solve uniquely for the h_{j}’s and the k_{j}^{r}’s. Furthermore, each power series in s terminates after finitely many terms, so that one gets polynomials of the desired form. (This was the method for computing (∗) in Example 2.6.) It is also clear that the coefficients of h and of k_{r} are (real) polynomials in the coefficients of p and the q_{r}, 1 ≤ r ≤ d.

For example, when d = 1, the formula becomes

exp (s (ip(t) + q \frac{d}{dt})) f(t)

= exp (i (sp(t) + \frac{qs^{2}}{2} p′(t) + \frac{q^{2}s^{3}}{3!} p′′(t) + \frac{q^{3}s^{4}}{4!} p′′′(t) + ⋯)) f(t + qs),

which appeared in Example 2.5. □

Now we are ready for the main result.

2.13. Theorem. Let Ω_{ε} be any stratum in Pedersen’s stratification of 𝔤^{∗}, and let 2d be the dimension of the orbits in that stratum. The affine variety Λ_{ε} parameterizes the representations corresponding to the G-orbits in Ω_{ε}, all of which are realized in L^{2}(ℝ^{d}). Then for any fixed x ∈ G, and for any pair f_{1} ∈ PW(ℝ^{d}), f_{2} ∈ C_{c}^{∞}(ℝ^{d}), the matrix coefficient

⟨π_{λ}(x)f_{1}, f_{2}⟩

extends to a holomorphic function of λ ∈ (Λ_{ε})_{ℂ}.

Proof. First we show that for f_{1} ∈ PW(ℝ^{d}), the function π_{λ}(x)f_{1}(t) extends to be holomorphic in t ∈ ℂ^{d}, x ∈ G_{ℂ}, and λ ∈ (Λ_{ε})_{ℂ}. To prove this, write

π_{λ}(x)f_{1}(t) = π_{λ}(exp x_{1}X_{1})π_{λ}(exp x_{2}X_{2}) ⋯ π_{λ}(exp x_{m}X_{m})f_{1}(t),

where x = (x_{1}, …, x_{m}) ∈ ℝ^{m} gives real analytic (in fact, polynomial) coordinates on G. We prove by descending induction on j that

π_{λ}(exp x_{j}X_{j}) ⋯ π_{λ}(exp x_{m}X_{m})f_{1}(t)

extends to be holomorphic in t ∈ ℂ^{d}, x_{j}, …, x_{m} ∈ ℂ, and λ ∈ (Λ_{ε})_{ℂ}. To start the induction, f_{1} is entire in t since it is in the Paley-Wiener class. The inductive step follows immediately from Proposition 2.11. To get the conclusion of the Theorem, observe that integration against f_{2} ∈ C_{c}^{∞}(ℝ^{d}) will (since f_{2} has compact support) yield a smooth function of x ∈ G and λ ∈ (Λ_{ε})_{ℂ}, holomorphic in λ. □

2.14. Remark. The proof of Theorem 2.13 reveals that the matrix coefficients of the operators arising from group elements are holomorphic, but unlike those arising from the enveloping algebra, they are not rational. As we saw in Example 2.5, it is also necessary to place further restrictions on f_{1}, f_{2} ∈ S(ℝ^{d}) in order for the corresponding matrix coefficient to be holomorphic.

2.15. Corollary. Let φ be a distribution on G with compact support. Then for any stratum Ω_{ε}, the operator-valued function

π_{λ}(φ) = ∫_{G} π_{λ}(x) dφ(x)

is weakly holomorphic on (Λ_{ε})_{ℂ}, in the sense that ⟨π_{λ}(φ)f_{1}, f_{2}⟩ is holomorphic for any pair f_{1} ∈ PW(ℝ^{d}), f_{2} ∈ C_{c}^{∞}(ℝ^{d}).

Proof. This is almost immediate from Theorem 2.13. For any pair f_{1} ∈ PW(ℝ^{d}), f_{2} ∈ C_{c}^{∞}(ℝ^{d}), the function

⟨π_{λ}(φ)f_{1}, f_{2}⟩ = ∫_{G}⟨π_{λ}(x)f_{1}, f_{2}⟩ dφ(x)

is the pairing of ⟨π_{λ}(x)f_{1}, f_{2}⟩, which is a smooth function of x and λ, holomorphic in λ for fixed x, against a distribution of compact support in x. Hence the result is still a holomorphic function of λ. □

Proof of Moss’s conjecture. We conclude the section now with the proof of Moss’s conjecture in the generality of an arbitrary simply connected nilpotent Lie group.

2.16. Theorem. Let G be a simply connected nilpotent Lie group. Suppose that φ is a distribution of compact support on G with the property that φ̂_{op} vanishes on a set of positive Plancherel measure in Ĝ. Then φ = 0.

Proof. We apply Corollary 2.15 to the top stratum Ω_{1}. The result is that φ̂_{op} is weakly holomorphic on (Λ_{1})_{ℂ}. Now Ω_{1} has full Lebesgue measure in 𝔤^{∗}, so Λ_{1} corresponds to a set of full Plancherel measure in Ĝ, and thus φ̂_{op} is, by the hypothesis of the theorem, zero on a set of positive measure in Λ_{1}. That can only happen if φ = 0. Indeed, any matrix coefficient of φ̂_{op} with respect to a function in PW(ℝ^{d}) and a function in C_{c}^{∞}(ℝ^{d}) will be a holomorphic function on (Λ_{1})_{ℂ} and zero on a set of positive measure in Λ_{1}. But the variety (Λ_{1})_{ℂ} is connected (by the same argument as in the proof of Theorem 1.1), and therefore the matrix coefficient must vanish on Λ_{1}. Since the spaces C_{c}^{∞}(ℝ^{d}) and PW(ℝ^{d}) are dense in L^{2}(ℝ^{d}), the operator φ̂_{op} itself is zero on Λ_{1}. Now by an application of the Plancherel Theorem on G we have that for any test function ψ ∈ D(G),

⟨φ, ψ⟩ = ∫_{Λ_{1}} Tr [φ̂_{op}(π)ψ̂_{op}(π)^{∗}] dμ(π)
= 0 ∀ψ.

That is, φ = 0. □

3. Vanishing of the Dixmier-Douady class for continuous-trace subquotients of the group C^{∗}-algebra of a nilpotent Lie group

Statement of the problem and the conjecture. In the introduction to this paper, we already indicated that the operator-valued Fourier transform of a function φ ∈ C_{c}^{∞} may be viewed as an element of the group C^{∗}-algebra C^{∗}(G). In general this is a very complicated object; for a nilpotent Lie group, about the most one knows about C^{∗}(G) is that it is a (generalized continuous-trace) liminary C^{∗}-algebra with a finite composition series into continuous-trace subquotients [Dix1]. Recall here that continuous-trace C^{∗}-algebras have Hausdorff spectrum and are the basic building blocks for all type I C^{∗}-algebras [Fell]. By the basic theory of Dixmier-Douady, a stable continuous-trace C^{∗}-algebra A, or in fact any ℵ_{0}-homogeneous continuous-trace C^{∗}-algebra A with finite-dimensional spectrum, is the algebra of sections of a locally trivial bundle of algebras over the locally compact Hausdorff space Â, with fibers isomorphic to the compact operators on an infinite-dimen-sional separable Hilbert space. Such a bundle is trivial if and only if its Dixmier-Douady invariant, which lies in H^{3}(Â, ℤ), vanishes. So to understand the operator-valued Fourier transform on a nilpotent Lie group, we necessarily encounter:

3.1. Question. Can C^{∗}(G) have a continuous-trace subquotient with non-zero Dixmier-Douady invariant?

If the answer is yes, then the non-trivial bundle structure of this subquotient is an essential feature in understanding the behavior of operator-valued Fourier transforms. If the answer is no, then it makes sense to understand the operator-valued Fourier transform (or at least its image in any continuous-trace subquotient) as an operator-valued function on Ĝ. In [RaRo, §4] and in [Ros, §3], we raised this question and conjectured the following:

3.2. Conjecture. If G is a connected nilpotent Lie group, every continuous-trace subquotient of C^{∗}(G) has vanishing Dixmier-Douady invariant.

We showed in [RaRo, §4] and in [Ros, §3] this is not the case for connected type I solvable Lie groups, nor is it the case for disconnected Lie groups in which the connected component of the identity is abelian and has finite index in G, but we gave some evidence for the truth of the corresponding conjecture for linear semi-simple Lie and 𝔭-adic groups.

Proof of the conjecture for two-step nilpotent Lie groups. Using an analysis related to our study in §1 of the Fourier transform for nilpotent Lie groups with “generically flat” orbits, we will now prove Conjecture 3.2 for two-step nilpotent Lie groups. The argument is interesting in that semi-simple Lie groups play an essential role in the proof.

3.3. Lemma. Let G be any two-step nilpotent Lie group with Lie algebra 𝔤. If A is a subquotient of C^{∗}(G) with Hausdorff spectrum, then there is no convergent sequence π_{n} → π in Â such that each π_{n} corresponds to an orbit O_{n} in 𝔤^{∗} of dimension 2r but such that π corresponds to an orbit O in 𝔤^{∗} of dimension different from 2r.

Proof. Recall that Ĝ is canonically homeomorphic to the “integral” part of 𝔤^{∗}/G. (“Integrality” is a vacuous condition if G is simply connected.) Now the orbits O_{n} cannot converge to an orbit of bigger dimension (since the subset of 𝔤^{∗} consisting of orbits of dimension ≤ 2r is Zariski-closed (cf. [Dix2, Proposition 1.11.5]) and thus closed in the Hausdorff topology), so we have to rule out the possibility that O could have strictly smaller dimension than 2r. However, if this is the case and π_{n} → π in Ĝ, then in fact the sequence {π_{n}} must have a whole affine space of limit points in Ĝ. (This is clear since all the G-orbits are flat; cf. Proposition 1.6.) Since Â is Hausdorff and is a locally closed subset of Ĝ, we get a contradiction. □

3.4. Theorem. Let G be any connected two-step nilpotent Lie group with Lie algebra 𝔤. If A is a subquotient of C^{∗}(G) with continuous trace, then A has vanishing Dixmier-Douady invariant (in other words, is Morita-equivalent to a commutative C^{∗}-algebra).

Proof. By Lemma 3.2, each connected component of Â consists only of representations corresponding to coadjoint orbits of some fixed dimension. So we may as well assume Â consists only of representations corresponding to coadjoint orbits of some fixed dimension 2r. (In fact in the two-step case, the orbits of fixed dimension 2r do indeed give a continuous-trace subquotient of C^{∗}(G), as pointed out in Corollary 6.3.4 of [Echt].)

Since the C^{∗}-algebra of any two-step nilpotent Lie group is a quotient of the C^{∗}-algebra of the universal two-step nilpotent Lie group G_{n} on some number n of generators (with Lie algebra 𝔤_{n} having a basis x_{1}, …, x_{n}, z_{ij} = [x_{i}, x_{j}], 1 ≤ i < j ≤ n, where the z_{ij}’s span the center 𝔷_{n}), it is enough to consider this case. Note that for any f ∈ 𝔤_{n}^{∗}, the alternating bilinear form B_{f} only depends on the restriction of f to 𝔷_{n}, and descends to an alternating form ω_{f} on 𝔤_{n}/𝔷_{n} ≅ ℝ^{n}. Also, knowledge of ω_{f} is equivalent to knowledge of f|_{𝔷_{n}}, since the matrix entries of ω_{f} are

(ω_{f})_{ij} = B_{f}(x_{i}, x_{j}) = f([x_{i}, x_{j}]) = \{ \begin{matrix}f(z_{ij}), & i < j, \\ 0, & i = j, \\ −f(z_{ji}), & i > j, \end{matrix}

and the z_{ij} are a basis for 𝔷_{n}. The orbit G_{n} · f is f + (rad B_{f})^{⊥}, whereas the set of functionals giving rise to the form ω_{f} is f + 𝔷_{n}^{⊥}. Thus ω_{f} only depends on the G-orbit of f, and the map G · f ↦ ω_{f} identifies the subspace Λ_{n, r} of 𝔤_{n}^{∗}/G_{n} consisting of orbits of dimension 2r with the total space of a vector bundle of rank n − 2r over the space X_{n, r} of skew-symmetric bilinear forms on ℝ^{n} of rank 2r, 1 ≤ r ≤ [\frac{n}{2}]. Since any such form is conjugate under the action of GL(n, ℝ) to the standard form given by

( \begin{matrix}0 & I_{r} & 0 \\ −I_{r} & 0 & 0 \\ 0 & 0 & 0 \end{matrix} ) ,

one finds that

X_{n, r} ≅ GL(n, ℝ)/ \{( \begin{matrix}A & B \\ 0 & D \end{matrix} ) : A ∈ Sp(2r, ℝ), D ∈ GL(n − 2r, ℝ), B arbitrary\} ,

which (if ≃ denotes “is homotopy-equivalent to”) gives

Λ_{n, r} ≃ X_{n, r} ≃ O(n)/U(r) × O(n − 2r) ≃ SO(n)/U(r) × SO(n − 2r).

This is a quotient of a connected compact Lie group by a connected subgroup of maximal rank, and thus is diffeomorphic to the quotient of a complex semisimple group by a parabolic subgroup, in other words, a generalized flag variety. It thus (by Bruhat decomposition) has a cell decomposition with only even-dimensional cells, and in particular its odd-dimensional cohomology vanishes. So H^{3}(X_{n, r}, ℤ) = 0 and the Dixmier-Douady invariant vanishes. □

The analysis in §1 of nilpotent Lie groups G having generic representations square-inte-grable modulo their projective kernels suggests that one can do a similar analysis to prove vanishing of the Dixmier-Douady invariant for the “large” continuous-trace ideal in C^{∗}(G).

References

[Ando] S. Ando, Paley-Wiener type theorem for Heisenberg groups, Proc. Japan Acad. 52 (1976), 331–333.

[ArnG] D. Arnal and S. Gutt, Décomposition de L^{2}(G) et transformation de Fourier adaptée pour un groupe G nilpotent, C. R. Acad. Sci. Paris Sér. I Math. 306 (1988), 25–28.

[Br] I. Brown, Dual topology of a nilpotent Lie group, Ann. Sci. École Norm. Sup. 6 (1973), 407–411.

[CorGr] L. Corwin and F. P. Greenleaf, Representations of nilpotent Lie groups and their Applications, Part 1: Basic Theory and Examples, Cambridge Studies in Advanced Math., vol. 18, Cambridge Univ. Press, Cambridge, New York, 1990.

[Dix1] J. Dixmier, Sur le dual d’un groupe de Lie nilpotent, Bull. Sci. Math. 90 (1966), 113–118.

[Dix2] J. Dixmier, Enveloping Algebras, North-Holland Math. Library, vol. 14, North-Holland, Amsterdam, New York, Oxford, 1977.

[Echt] S. Echterhoff, Crossed products with continuous trace, Habilitationsschrift, Universität-Gesamt-hochschule Paderborn, Paderborn, 1993.

[Fell] J. M. G. Fell, The structure of algebras of operator fields, Acta Math. 106 (1961), 233–280.

[Hel] S. Helgason, The Radon Transform, Progress in Math., vol. 5, Birkhäuser, Boston, Basel, Stuttgart, 1980.

[Joy] K. T. Joy, A description of the topology on the dual space of a nilpotent Lie group, Pacific J. Math. 112 (1984), 135–139.

[Kum] K. Kumahara, An analogue of the Paley-Wiener Theorem for the Heisenberg group, Proc. Japan Acad. 47 (1971), 491–494.

[MoorWo] C. C. Moore and J. A. Wolf, Square integrable representations of nilpotent groups, Trans. Amer. Math. Soc. 185 (1973), 445–462.

[Moss] J. D. Moss, Jr., A Paley-Wiener Theorem for selected nilpotent Lie groups, J. Funct. Anal. 114 (1993), 395–411.

[Mum] D. Mumford, Algebraic Geometry, I: Complex Projective Varieties, Grundlehren der math. Wiss., vol. 221, Springer-Verlag, Berlin, Heidelberg, New York, 1976.

[Park] R. Park, A Paley-Wiener Theorem for all two- and three-step nilpotent Lie groups, Preprint (1994).

[Ped1] N. V. Pedersen, On the symplectic structure of coadjoint orbits of (solvable) Lie groups and applications, I, Math. Ann. 281 (1988), 633–669.

[Ped2] N. V. Pedersen, Geometric quantization and the universal enveloping algebra of a nilpotent Lie group, Trans. Amer. Math. Soc. 315 (1989), 511–563.

[Ped3] N. V. Pedersen, Geometric quantization and nilpotent Lie groups: a collection of examples, Univ. of Copenhagen, Copenhagen, 1988.

[Ped4] N. V. Pedersen, Orbits and primitive ideals of solvable Lie algebras, Math. Ann. 298 (1994), 275–326.

[Puk] L. Pukanszky, On Kirillov’s character formula, J. reine ang. Math. 311 (1979), 408–440.

[RaRo] I. Raeburn and J. Rosenberg, Crossed products of continuous-trace C^{∗}-algebras by smooth actions, Trans. Amer. Math. Soc. 305 (1988), 1–45.

[Ros] J. Rosenberg, C^{∗}-algebras and Mackey’s theory of group representations, C^{∗}-algebras: 1943– 1993, a Fifty Year Celebration (R. S. Doran, ed.), Contemp. Math., vol. 167, Amer. Math. Soc., Providence, RI, 1994, pp. 151–181.

[ScSi] D. Scott and A. Sitaram, Some remarks on the Pompieu problem for groups, Proc. Amer. Math. Soc. 104 (1988), 1261–1266.

[Sev1] L. A. Sevastyanov, The Paley-Wiener Theorem for the Heisenberg group, Trudy Univ. Druzby Narod. 80 (1976), no. 9, 122–126; MR 58 #28277.

[Sev2] L. A. Sevastyanov, An analogue of the Paley-Wiener Theorem for a nilpotent Lie group, An analogue of the Paley-Wiener Theorem for metabelian Lie groups, Topological Spaces and Their Mappings, Latv. Gos. Univ. Riga, Riga, 1981, pp. 130–136, 178, 184–185; MR 84a:22024.

[Shaf] I. Shafarevich, Basic Algebraic Geometry, transl. by K. A. Hirsch, Grundlehren der math. Wiss., vol. 213, Springer-Verlag, Berlin, Heidelberg, New York, 1974 and 1977.

Department of Mathematics, University of Maryland, College Park, MD 20742

   E-mail address: rll@math.umd.edu; jmr@math.umd.edu