% ------------------------------------------------------------------------ % AMS-LaTeX Paper ******************************************************** % ------------------------------------------------------------------------ % Submitted: Trans.Amer.Math.Soc. in February 1995 % Final Version: July 1995 % Accepted: June 1995 % ------------------------------------------------------------------------ % This is a journal top-matter template file for use with AMS-LaTeX. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \documentclass{tran-l} %\usepackage[active]{srcltx} % SRC Specials for DVI Searching % Over-full v-boxes on even pages are due to the \v{c} in author's name \vfuzz2pt % Don't report over-full v-boxes if over-edge is small % THEOREM Environments --------------------------------------------------- \newtheorem{thm}{Theorem}[subsection] \newtheorem{cor}[thm]{Corollary} \newtheorem{lem}[thm]{Lemma} \newtheorem{prop}[thm]{Proposition} \theoremstyle{definition} \newtheorem{defn}[thm]{Definition} \theoremstyle{remark} \newtheorem{rem}[thm]{Remark} \numberwithin{equation}{subsection} % MATH ------------------------------------------------------------------- \DeclareMathOperator{\RE}{Re} \DeclareMathOperator{\IM}{Im} \DeclareMathOperator{\ess}{ess} \newcommand{\eps}{\varepsilon} \newcommand{\To}{\longrightarrow} \newcommand{\h}{\mathcal{H}} \newcommand{\s}{\mathcal{S}} \newcommand{\A}{\mathcal{A}} \newcommand{\J}{\mathcal{J}} \newcommand{\M}{\mathcal{M}} \newcommand{\W}{\mathcal{W}} \newcommand{\X}{\mathcal{X}} \newcommand{\BOP}{\mathbf{B}} \newcommand{\BH}{\mathbf{B}(\mathcal{H})} \newcommand{\KH}{\mathcal{K}(\mathcal{H})} \newcommand{\Real}{\mathbb{R}} \newcommand{\Complex}{\mathbb{C}} \newcommand{\Field}{\mathbb{F}} \newcommand{\RPlus}{\Real^{+}} \newcommand{\Polar}{\mathcal{P}_{\s}} \newcommand{\Poly}{\mathcal{P}(E)} \newcommand{\EssD}{\mathcal{D}} \newcommand{\Lom}{\mathcal{L}} \newcommand{\States}{\mathcal{T}} \newcommand{\abs}[1]{\left\vert#1\right\vert} \newcommand{\set}[1]{\left\{#1\right\}} \newcommand{\seq}[1]{\left<#1\right>} \newcommand{\norm}[1]{\left\Vert#1\right\Vert} \newcommand{\essnorm}[1]{\norm{#1}_{\ess}} %%% ---------------------------------------------------------------------- \begin{document} \title[An Extension of Lomonosov's Techniques] {An Extension of Lomonosov's Techniques to non-compact Operators} \author{ Aleksander Simoni\v{c} } \address{Department of Mathematics, Statistics \& Computing Science, Dalhousie University, Halifax, Nova Scotia, B3H~3J5, Canada} \email{alex@winedt.com} \thanks{This work was completed with the support of an Izaak Walton Killam Memorial Scholarship.} \thanks{The author was also supported in part by the Research Council of Slovenia.} \subjclass{Primary 47A15; Secondary 46A32, 47D20} \keywords{linear operator, invariant subspace, transitive algebra} \date{February 15, 1995 and, in revised form, July 6, 1995.} \dedicatory{} \commby{Daniel J. Rudolph} %%% ---------------------------------------------------------------------- \begin{abstract} The aim of this work is to generalize Lomonosov's techniques in order to apply them to a wider class of not necessarily compact operators. We start by establishing a connection between the existence of invariant subspaces and density of what we define as the associated Lomonosov space in a certain function space. On a Hilbert space, approximation with Lomonosov functions results in an extended version of Burnside's Theorem. An application of this theorem to the algebra generated by an essentially self-adjoint operator $A$ yields the existence of vector states on the space of all polynomials restricted to the essential spectrum of $A$. Finally, the invariant subspace problem for compact perturbations of self-adjoint operators acting on a real Hilbert space is translated into an extreme problem and the solution is obtained upon differentiating certain real-valued functions at their extreme. \end{abstract} %%% ---------------------------------------------------------------------- \maketitle %%% ---------------------------------------------------------------------- \section*{Introduction} The question \begin{quote} \emph{Does every bounded linear operator on a Banach space have a non-trivial closed invariant subspace?} \end{quote} is known as the \emph{invariant subspace problem}. \smallskip Examples due to Enflo~\cite{Enf87} and Read~\cite{Rea85} show that the answer to the invariant subspace problem is in general negative. However, there are no known examples of operators without invariant subspaces acting on a reflexive Banach space and in particular, on a Hilbert space. Furthermore, there seems to be no evidence what should be an expected answer for the operators acting on a Hilbert space, and the experts in the field have different opinions on it. It is therefore not surprising that there are relatively few special cases for which the existence of invariant subspaces have been established. One of the most prominent results is the one on the existence of hyperinvariant subspaces for compact operators due to V.I.~Lomonosov~\cite{Lom73,RR73}. Another class of operators that is well understood in terms of invariant subspaces are normal, and in particular, self-adjoint operators for which there is the powerful spectral theorem. However, it is not known whether a compact perturbation of a self-adjoint operator has a non-trivial invariant subspace~\cite{Lom92}. This paper focuses on the existence of invariant subspaces for essentially self-adjoint operators and culminates in an affirmative answer in the case when the underlying Hilbert space is assumed to be \emph{real}. When dealing with the existence of invariant subspaces it is a common practice~\cite{AAB95,dB93,Lom91,Sim96a} to study the space of certain continuous functions associated with the algebra generated by an operator rather than the operator itself. We follow this approach and establish a connection between the existence of invariant subspaces for an operator algebra and density of certain associated spaces of continuous functions called \emph{Lomonosov spaces}. The construction of these functions is based on the idea of the \emph{partition of unity} subordinate to an open cover, which is a standard tool in approximation theory~\cite{Gam90} and differential geometry~\cite{Dev68,Spi65}. In~\cite{Dev68} the partition of unity is also used to prove the Arzela-Ascoli Theorem. It should be observed that a similar argument was employed by V.I.~Lomonosov in the proof of his celebrated 1970's result on compact operators~\cite{Lom73}. On a Hilbert space differentiability of the norm yields a numerical criterion for the construction of Lomonosov functions with certain properties. The Schauder-Tychonoff Fixed Point Theorem~\cite[p.~456]{DS57}, applied to certain Lomonosov functions, yields an extension of the Burnside Theorem and gives the solution of what we define as the ``essentially-transitive algebra problem''. An application of the extended Burnside Theorem to the algebra generated by an essentially self-adjoint operator implies the existence of vector states on the space of polynomials restricted to the essential spectrum of such an operator. The invariant subspace problem for compact perturbations of self-adjoint operators is translated into an extreme problem and the solution (in the case where the underlying Hilbert space is real) is obtained upon differentiating certain real-valued functions at their extreme. Although the above-described techniques do not immediately extend to the complex Hilbert spaces, it is very likely that further analysis of the space of vector states will reduce the complex case to the real one and thus provide the affirmative answer to one of the most difficult questions in the theory of invariant subspaces~\cite{Lom92}. %%% ---------------------------------------------------------------------- \section{An Extension of Burnside's Theorem} In this section we combine differentiability of the Hilbert norm with the Schauder-Tychonoff Fixed Point Theorem to show that every closed subalgebra $\A\subset\BH$, acting on a real or complex Hilbert space $\h$, either contains a nonzero finite-rank operator, or there exist nonzero vectors $f,g\in\h$, such that for all $A\in\A$: \[ \abs{\RE\seq{A f,g}}\leq\essnorm{\RE A}\seq{f,g}. \] This result generalizes an extension of Burnside's Theorem, recently obtained by V.I.\,Lomonosov, using rather different techniques. The theory developed in this section has an interesting application to the invariant subspace problem for essentially self-adjoint operators which is given at the end of this paper. %%% ---------------------------------------------------------------------- \subsection{Introduction} In~\cite{Sim96a} we define the \emph{Lomonosov space} and give a constructive proof of the approximation theorem inspired by the well known result of V.I.~Lo\-mo\-no\-sov \cite{Lom73}. This theorem is then applied to establish a connection between the existence of invariant subspaces for the norm dual of an algebra of bounded operators on a Banach space, and density of the associated Lomonosov space in certain function spaces. These results cover recent characterizations of the invariant subspace problem by Y.A.~Abramovich, C.D.~Aliprantis, and O.~Burkinshaw in~\cite{AAB95}, who obtained their results using rather different techniques introduced in~\cite{Lom91} and further exploited in~\cite{dB93}. In this section we combine differentiability of the Hilbert norm with a construction of Lomonosov functions and the Schauder-Tychonoff Fixed Point Theorem to establish a connection between the Lomonosov space and the transitive algebra problem~\cite{RR73}. We start by briefly introducing a simplified Hilbert space terminology, that is consistent with~\cite{Sim96a}, where the corresponding terms are defined in a more general Banach space setting. \begin{defn} Suppose $\s$ is a bounded closed convex subset of a real or complex Hilbert space $\h$, equipped with the relative weak topology. The set of all continuous functions $f\colon\s\To\h$, where both spaces are equipped with the weak topology, is denoted by $C(\s,\h)$. Similarly, $C(\s,[0,1])$ stands for the set of all weakly-continuous functions $f\colon\s\To[0,1]$. \end{defn} \begin{rem} Recall that a bounded closed convex subset $\s$ in a Hilbert space is weakly compact. Observe also that a bounded linear operator $A\in\BH$ is in $C(\s,\h)$. Whenever we say that $A$ is in $C(\s,\h)$, we actually refer to the restriction of the operator $A\in\BH$ to the subset $\s\subset\h$. \end{rem} \begin{defn} Let $\A$ be a subset of $C(\s,\h)$. The convex set $\Lom(\A) \subset C(\s,\h)$, defined by \[ \Lom(\A) = \set{ \sum_{k=1}^n \alpha_k A_k\,\,|\,\,\, A_k\in\A,\, \alpha_k\in C(\s,[0,1]) \text{ and } \sum_{k=1}^n\alpha_k \equiv 1; \,\, n < \infty}, \] is called the \emph{Lomonosov space} associated with the set $\A$, and a function $\Lambda\in\Lom(\A)$ is called a \emph{Lomonosov function}. \end{defn} \begin{defn} Let $\W$ be a basic weak neighborhood of a vector $f\in\h$: \begin{equation}\label{e:WeakBase} \W=\set{g\in\h\,\,|\,\,\,\abs{\seq{f-g,h_k}} < 1, ~~ h_k\in\h, \quad k=1,\ldots,n; \,\, n < \infty}. \end{equation} A continuous nonnegative function $\Gamma_\W\colon\h\To[0,1]$, defined by \begin{equation}\label{e:CIF} \Gamma_\W(g)= \prod_{k=1}^n \max\set{0,1-\abs{\seq{f-g,h_k}}}, \end{equation} is called a \emph{continuous indicator function} of $\W$. \end{defn} \begin{rem} Clearly, $\Gamma_\W$ is a nonnegative weakly continuous function, and \[ \W = \set{g\in\h\,\,|\,\,\,\Gamma_\W(g)>0}. \] \end{rem} The following proposition and its corollary introduce the idea that will lead to the main result of this section. \begin{prop} \label{p:wAPPROX} Let $\s$ be a closed bounded and convex subset of $\h$. Suppose the set $\A\subset{C}(\s,\h)$ satisfies the following property: \begin{quote} {For every $s\in\s$ there exists a function $A_s\in\A$, together with a weak neighborhood $\W_s$ of $s$, such that $A_s(\W_s)\subset\s$.} \end{quote} Then there exists a Lomonosov function $\Lambda\in\Lom(\A)$ that maps the set $\s$ into itself. \end{prop} \begin{proof} By the hypothesis for every point $s\in\s$ there exists a function $A_s$, together with a basic weak neighborhood $\W_s$ of $s$, so that $A_s(\W_s)\subset\s$. In this way we obtain an open cover of $\s$. Since $\s$ is a weakly compact set there exists a finite subcover $\W_1,\ldots,\W_n$, together with functions $A_1,\ldots,A_n$, with the property that $A_k(\W_k)\subset\s$ for $k=1,\ldots,n$. Let $\Gamma_k\colon\s\To[0,1]$ denote the continuous indicator function of $\W_k$ as defined by~(\ref{e:CIF}). Each point $s\in\s$ lies at least in one neighborhood $\W_k$ $(k=1,\ldots,n)$; therefore the sum $\sum_{j=1}^n\Gamma_j(s)$ is strictly positive for all vectors $s\in\s$. Hence, the functions $\alpha_k\colon\s\To[0,1]$, defined by \[ \alpha_k(s) = \frac{\Gamma_k(s)}{\sum_{j=1}^n \Gamma_j(s)} \quad\quad (k=1,\ldots,n), \] are well defined and weakly continuous on $\s$. Also, $\sum_{k=1}^n\alpha_k(s)=1$ for every $s\in\s$, and $\alpha_k(s)>0$ if and only if $s\in\W_k$. The Lomonosov function $\Lambda\colon\s\To\s$, in the Lomonosov space $\Lom(\A)$, associated with the set of functions $\A\subset{C}(\s,\h)$, is defined by \[ \Lambda(s) = \sum_{k=1}^n \alpha_k(s) A_k(s). \] Observe that $\Lambda(s)$ is a convex combination of the elements in $\s$, and consequently, $\Lambda$ maps the set $\s$ into itself (weak-continuously). \end{proof} \begin{cor} \label{c:wAPPROX} Suppose $\A$ is a convex subset of $C(\s,\h)$ satisfying the condition of Proposition~\ref{p:wAPPROX}. Then there exists an element $A\in\A$ with a fixed point $s\in\s$. \end{cor} \begin{proof} By the Schauder-Tychonoff Fixed Point Theorem~\cite[p.~456]{DS57} the Lomonosov function $\Lambda\colon\s\To\s$, constructed in the proof of Proposition~\ref{p:wAPPROX}, has a fixed point $s\in\s$. Let $A=\sum_{k=1}^n\alpha_k(s)A_k$. Convexity of the set $\A$ implies that $A\in\A$. Furthermore, from $\Lambda(s)=s$ it follows that $A(s)=s$. \end{proof} \begin{rem} In our applications we consider the situations when $\s$ is a closed ball of radius $r\in(0,1)$ around a fixed unit vector $f_0\in\h$, and $\A$ is a convex subset of $\BH$. If the set $\A$ satisfies the condition of Proposition~\ref{p:wAPPROX} then Corollary~\ref{c:wAPPROX} implies that the set $\A$ contains an operator $A$ with an eigenvalue $1$. This gives rise to the following two questions: \begin{itemize} \item{When does the set $\A$ satisfy the condition of Proposition~\ref{p:wAPPROX}?} \item{When is the operator $A$ in Corollary~\ref{c:wAPPROX} different from the identity?} \end{itemize} Complete continuity of compact operators, restricted to $\s$, yields an affirmative answer to the first question whenever $\A$ is a set of compact operators with the property that for every vector $s\in\s$ there exists an operator $A_s\in\A$ such that $\norm{A_s{s}-f_0} \essnorm{\RE{A}}. \end{equation} Then the norm closure of the real algebra generated by $A$ contains a nonzero finite-rank operator. \end{prop} \begin{proof} We may assume that the Hilbert space $\h$ is complex as long as we can construct a finite-rank operator in the closure of the \emph{real} algebra generated by $A$. Clearly, (\ref{e:ESSINEQ}) implies that $\lambda$ is not in the essential spectrum of $A$. From the well known properties of the essential spectrum (for example, Theorem~6.8 and Proposition~6.9 in~\cite[p.~366]{Con90}), it follows that every point in the spectrum of the operator $A$, satisfying the condition~(\ref{e:ESSINEQ}), is an isolated eigenvalue of $A$, and the corresponding Riesz projection has finite rank. After first replacing the operator $A$ by $-A$ in the case when $\RE\lambda<0$, and then substituting the translation $A-\max\set{\RE\lambda\,\,|\,\,\,\lambda\in\sigma(A)}$ for $A$, we may assume that $\max_{\lambda\in\sigma(A)}{\RE\lambda}=0$. The condition (\ref{e:ESSINEQ}) implies that \[ \sigma_0(A)=\set{\lambda\in\sigma(A)\,\,|\,\,\,\RE\lambda=0}, \] is a nonempty finite set, consisting of eigenvalues of $A$ with finite multiplicity. By the Riesz Decomposition Theorem~\cite[p.~31]{RR73}, the space $\h$ can be decomposed as $\h=\h_1\oplus\h_2$, where $\dim{\h_1}<\infty$, and $A$ is similar to the operator $A_1\oplus{A_2}$, corresponding to this decomposition. Furthermore, the spectrum of $A_1$ is $\sigma_0(A)$, and the spectrum of $A_2$ lies in the open left complex half-plane. Therefore $r(e^{t{A_2}})<1$ for $t>0$, while $r(e^{t{A_1}})=1$ for any real argument $t$. By Rota's Theorem~\cite[p.~136]{Pau86}, the operator $e^{A_2}$ is similar to a strict contraction. Consequently, \[ \lim_{n\To\infty}\frac{ e^{n{A_2}} }{ \norm{e^{n{A_1}}} }=0. \] On the other hand, finite-dimensionality of $\h_1$ implies that the sequence \[ T_n = \frac{e^{n{A_1}}}{\norm{e^{n{A_1}}}}, ~ \qquad n=0,1,\ldots, \] has a subsequence converging in norm to a nonzero finite-rank operator. \end{proof} \begin{rem} Recall that the exponential function $e^A$ admits the power series: \[ e^A = \sum_{n=0}^{\infty} \frac{A^n}{n!}. \] Hence, the finite-rank operator constructed in the proof of Proposition~\ref{p:ESSPECT} is indeed contained in the norm closure of the \emph{real} algebra, generated by $A$. \end{rem} %%% ---------------------------------------------------------------------- \subsection{Preliminary Geometric Results} This subsection contains preliminary results that are needed in the constructive proof of the main theorem, given in the next subsection. The results presented here are mostly easy observations and the proofs are somehow tedious and straightforward calculations, involving the standard ``$\eps,\delta$'' arguments. Throughout this subsection we make the following conventions: Let $\h$ be a real or complex Hilbert space. Fix a unit vector $f_0\in\h$ and choose a positive number $r\in(0,1)$. The set $\s$ is defined as the ball of radius $r$ around the vector $f_0$: \[ \s = \set{f\in\h\,\,|\,\,\, \norm{f_0-f}\leq r}. \] \begin{lem} \label{l:APPROX} Let $\W$ be a subset of $\s$ and let $A$ be a bounded linear operator on $\h$. Suppose that \[ \RE\seq{A{f},f_0-f}\geq\delta>0, \quad \text{\em for all $f\in\W$.} \] Then there exists a positive number $\mu>0$, such that for any $\eps\in(0,\mu)$: \[ \norm{f_0 - (I+\eps A)f} < \norm{f_0 - f}, \quad \text{\em for all $f\in\W$.} \] \end{lem} \begin{proof} Note that $f\in\s$ implies $\norm{f} \leq 1 + r < 2$. Therefore for all $f\in\s$: \[ \norm{A{f}} \leq \norm{A}\norm{f} \leq (1+r) \norm{A} < 2\norm{A}. \] Set $\mu=\tfrac{\delta}{2\norm{A}^2}$. For any $\eps\in(0,\mu)$ and $f\in\W$ we have: \begin{align*} \norm{f_0 - (I+\eps A)f}^2 &= \norm{f_0-f - \eps A f}^2 \\ & = \norm{f_0-f}^2 - 2\eps\RE\seq{A{f},f_0-f} + \eps^2\norm{A{f}}^2 \\ & < \norm{f_0-f}^2 - 2\eps\delta + \eps^2 4\norm{A}^2 \\ & = \norm{f_0-f}^2 - 2\eps(\delta - \eps 2\norm{A}^2) \\ & < \norm{f_0-f}^2. \end{align*} Hence, $\mu$ is the required positive number. \end{proof} \begin{rem} Let $\psi'(0)$ denote the derivative of the function \[ \psi(t)=\norm{(I+t{A})f-f_0}^2, \] with respect to $t$, at the point $t=0$. A straightforward calculation yields: \[ \psi'(0)=-2\RE\seq{A{f},f_0-f}. \] Therefore the statement of Lemma~\ref{l:APPROX} corresponds to the well known fact that a real function with (strictly) negative derivative is (strictly) decreasing. Note, however, that positivity of $\RE\seq{A{f},f_0-f}$ does not imply that the mapping $\Psi(f)=(I+\eps A)f$ is a contraction, as a function from $\W$ to $\s$. \end{rem} Lemma~\ref{l:APPROX} gives a numerical criterion for the subset $\W\subset\s$ to be mapped into $\s$, namely positivity of the function $\Phi(f)=\RE\seq{A{f},f_0-f}$ on $\W$. Since $\Phi(f_0)=0$, this criterion cannot be employed at the point $f_0$. However, the construction of a function $\Lambda\colon\s\To\s$ can be easily restricted to the subset of $\s$ not containing the point $f_0$. A simple observation in $\Real^2$ suggests the following definition. \begin{defn} For a fixed ball $\s=\set{f\in\h\,\,|\,\,\, \norm{f_0-f}\leq r}$ around the unit vector $f_0\in\h$ the \emph{polar hyperplane} $\Polar$, of the origin with respect to $\s$, is defined by the following set: \[ \Polar = \set{f\in\h\,\,|\,\,\, \seq{f,f_0}=1-r^2}. \] \end{defn} \begin{rem} Every vector $f$ in $\Polar\cap\s$ can be decomposed as\, $ f = (1-r^2)f_0 + g$, where $g\perp f_0$ and $\norm{g}^2\leq r^2(1-r^2)$. In particular, the boundary of $\Polar\cap\s$: \[ \Polar \cap \partial \s = \set{(1-r^2)f_0 + g\,\,\big|\,\,\, g\perp f_0 \quad\text{ and }\quad \norm{g}^2=r^2(1-r^2)}, \] contains exactly the points where the tangents from the origin to the ball $\s$ intersect $\s$. Recall that in $\Real^2$ such a line is called a \emph{polar},\, and our definition is just a straightforward generalization of this geometric term to higher dimensional Hilbert spaces. \end{rem} The following lemma will ``restrict'' the construction of a Lomonosov function $\Lambda\colon\s\To\s$ to the polar hyperplane $\Polar$. \begin{lem} \label{l:SCALE} The function $\Lambda_0\colon \s \To \s$, defined by \[ \Lambda_0(f) = \frac{1}{r^2+\seq{f,f_0}} f, \] maps the set $\s=\set{f\in\h\,\,|\,\,\, \norm{f-f_0}\leq{r}}$ weak-continuously into itself. Furthermore, the set of all fixed points for $\Lambda_0$ is equal to $\Polar\cap\s$. \end{lem} \begin{proof} Since $\RE\seq{f,f_0}>0$ for $f\in\s$, it follows that $\Lambda_0$ is well defined and weakly continuous on $\s$. Clearly, $f\in\s$ is a fixed point for $\Lambda_0$ if and only if $r^2+\seq{f,f_0}=1$. By the definition of the polar hyperplane, that is equivalent to $f\in\Polar\cap\s$. We have to prove that $\norm{\Lambda_0(f)-f_0}\leq r$ for all $f\in\s$. Every vector $f\in\s$ can be decomposed as $f=\seq{f,f_0}f_0+g$, where $g \perp f_0$ and $\norm{g}^2\leq r^2-\abs{1-\seq{f,f_0}}^2$. \goodbreak A straightforward calculation, using this decomposition, yields: \begin{align*} \norm{\Lambda_0(f)-f_0}^2 &= \norm{\frac{1}{r^2+\seq{f,f_0}} f - f_0}^2\\ &= \frac{1}{\abs{r^2+\seq{f,f_0}}^2} \left(r^4+\norm{g}^2\right)\\ &\leq \frac{1}{\abs{r^2+\seq{f,f_0}}^2} \left(r^4+r^2-\abs{1-\seq{f,f_0}}^2\right). \end{align*} The conclusion follows if we can establish the following inequality: \[ r^4+r^2 - \abs{1-\seq{f,f_0}}^2 \leq r^2\abs{r^2+\seq{f,f_0}}^2. \] Setting $\seq{f,f_0}=x+i{y}$, this can be translated to \[ r^4+r^2 - (1-x)^2 - y^2 \leq r^2(r^2 + x)^2 + r^2y^2, \] or, equivalently: \begin{align*} (1+r^2)y^2 &\geq r^4+r^2 - (1-x)^2 - r^2(r^2 + x)^2\\ & = (r^4+r^2-1-r^6) + 2(1-r^4)x - (1+r^2)x^2 \\ & = -(1+r^2)\big((1-r^2) - x\big)^2. \end{align*} The last inequality is obviously always satisfied, with the strict inequality holding everywhere, except in the polar hyperplane $\Polar$. \end{proof} \begin{defn} For $A\in\BH$ define a real-valued function $\Delta_A\colon\s\To\Real$ as follows: \[ \Delta_A(f) = \frac{1}{r^2(1-r^2)}\RE\seq{A{f}, f_0-f}. \] \end{defn} \begin{rem} Note that $\Delta_A$ is a ``normalization'' of the function $\RE\seq{A{f},f_0-f}$ in Lemma~\ref{l:APPROX}. From the definition of the set $\Polar$ it follows that every vector $f$ in $\Polar\cap\s$ can be decomposed as $f=(1-r^2)f_0 + r\sqrt{1-r^2} g$, where $g \perp f_0$ and $\norm{g}\leq 1$. Consequently, \begin{equation}\label{e:NUMERICAL} \begin{split} \Delta_A(f) &= \RE\seq{A\left(f_0+\tfrac{r}{\sqrt{1-r^2}}g\right), f_0-\tfrac{\sqrt{1-r^2}}{r}g}\\ &= \RE\seq{A f_0, f_0} - \RE\seq{Ag, g} - \RE\seq{\left(\tfrac{\sqrt{1-r^2}}{r}A -\tfrac{r}{\sqrt{1-r^2}}A^*\right) f_0, g}. \end{split} \end{equation} In particular, for the identity operator $I$ on $\h$, we have $\Delta_I(f)=1-\norm{g}^2$. Therefore, $\Delta_I\geq0$ on $\Polar\cap\s$, with the equality $\Delta_I(f)=0$ holding if an only if $f\in\Polar\cap\partial\s$. Observe that the function $\Delta_A\colon\s\To\Real$ is norm continuous, but it is in general not weakly continuous, due to the presence of the quadratic form: \[ \RE\seq{A g,g} = \seq{(\RE{A})g,g} \] in~(\ref{e:NUMERICAL}). \end{rem} \begin{lem} \label{l:EssNR} Suppose $\delta$ and $M$ are positive numbers, and $A$ is a fixed operator in $\BH$. Then there exists a weak neighborhood $\W$ of $0$ in $\h$ such that every vector $f\in\W$, with $\norm{f}\leq{M}$, satisfies the following inequality: \[ \abs{\RE\seq{A{f},f}} < \essnorm{\RE{A}}\norm{f}^2 + \delta. \] \end{lem} \begin{proof} From $\RE\seq{A f,f}=\seq{(\RE{A})f,f}$ and $(\RE{A})=(\RE{A})^*$ it follows that \[ \norm{\RE{A}} = \sup_{f\neq0} \abs{\RE\seq{A f,f}}\norm{f}^{-2}. \] By definition of the essential norm, we have: \[ \essnorm{\RE A} = \inf_{K\in\KH} \norm{(\RE A)+K} = \inf_{K\in\KH} \norm{\RE(A+K)}. \] Hence, there exists a compact operator $K$ such that \[ \essnorm{\RE A} > \norm{\RE(A+K)} - \tfrac{1}{2}\delta M^{-2} \geq \abs{\RE\seq{(A+K)f,f}}\norm{f}^{-2} - \tfrac{1}{2}\delta M^{-2} . \] The proposition now follows by the mixed (weak-to-norm) continuity of compact operators on bounded sets. \end{proof} The next lemma imposes a condition on the operator $A$ that guarantees the existence of a weak neighborhood of $f$ in $\s$ on which $\Delta_A$ is positive. \begin{lem} \label{l:ESS} Suppose $f$ is a vector in the polar hyperplane $\Polar\cap\s$, satisfying the following \emph{strict} inequality for some $A \in \BH$: \[ \Delta_A(f) > \essnorm{\RE{A}} \Delta_I(f). \] Then there exists a positive number $\delta>0$, together with a weak neighborhood $\W$ of $f$, such that for every $h\in\W\cap\s$: \[ \Delta_A(h) > \essnorm{\RE{A}}\abs{\Delta_I(h)} + \delta. \] \end{lem} \begin{proof} By the hypothesis, there exists a positive number $\delta>0$ such that: \begin{equation}\label{e:EC1} \Delta_A(f) > \essnorm{\RE{A}}\Delta_I(f) + 5\delta. \end{equation} For any positive number $0<\eps \frac{(1-r^2-\eps)(r^2-\eps)-r^2+(r^2-\eps)^2}{r^2(1-r^2)} \\ &> -\frac{3\eps}{r^2(1-r^2)}. \end{align*} Therefore, a weak neighborhood $\W_\eps$ of $\Polar$, such that $\essnorm{\RE{A}} \Delta_I(h) > -\delta$, for every vector $h\in\W_\eps\cap\s$, can be obtained by setting \[ \eps=\frac{\delta{r^2}(1-r^2)}{1+3\essnorm{\RE{A}}}. \] A straightforward calculation yields: \begin{equation}\label{e:EC2} \Delta_A(f+g) = \Delta_A(f) + \tfrac{1}{r^2(1-r^2)} \RE\big(\seq{A{g},f_0-f} - \seq{A{f},g} - \seq{A{g},g} \big). \end{equation} Lemma~\ref{l:EssNR} implies the existence of a weak neighborhood $\W_1$ of $0$, such that for every vector $g\in\W_1$, with $\norm{g}\leq2$: \begin{equation}\label{e:EC3} \RE\seq{A g,g} < \essnorm{\RE{A}} \norm{g}^2 + r^2(1-r^2)\delta. \end{equation} Clearly, by the weak-continuity of both sides of the inequality, there exists a weak neighborhood $\W_2$ of $0$ such that for $g\in\W_2$: \begin{equation}\label{e:EC4} \begin{split} \RE\seq{A g,f_0-f} & - \RE\seq{A f,g} > \\ & \essnorm{\RE{A}}\big(\RE\seq{g,f_0-f}-\RE\seq{f,g}\big) -r^2(1-r^2)\delta. \end{split} \end{equation} Let $\W = \W_\eps\cap\big(f + \W_1\cap\W_2\big)$ be a weak neighborhood of $f$. Every vector $h$ in $\W\cap\s$ can be written as $h=f+g$, where $g \in \W_1\cap\W_2$ and $\norm{g}<2$. Putting the inequalities (\ref{e:EC1}\,--\,\ref{e:EC4}) together, and using $\essnorm{\RE{A}}\Delta_I(h)>-\delta$, implies: \begin{align*} \Delta_A(h) &= \Delta_A(f+g) \\ &= \Delta_A(f) + \tfrac{1}{r^2(1-r^2)} \RE( \seq{A g,f_0-f} - \seq{A f,g} - \seq{A g,g})\\ &> \essnorm{\RE{A}}\big( \Delta_I(f) + \tfrac{1}{r^2(1-r^2)}\RE (\seq{g,f_0-f} - \seq{f,g} - \norm{g}^2)\big) + 3\delta\\ &= \essnorm{\RE{A}} \Delta_I(f+g) + 3\delta\\ &= \essnorm{\RE{A}} \Delta_I(h) + 3\delta\\ &> \essnorm{\RE{A}} \abs{\Delta_I(h)} + \delta. \end{align*} Consequently,\, $\W$ is a weak neighborhood of $f$ with the required property. \end{proof} %%% ---------------------------------------------------------------------- \subsection{The Main Result} We are now ready to give the main result of this section, which is quite technical, but applicable to several situations discussed later. \begin{prop} \label{p:ESS} Let $\A\subset\BH$ be a convex subset of bounded linear operators acting on a real or complex Hilbert space $\h$. Fix a unit vector $f_0\in\h$ and choose a positive number $r\in(0,1)$. Suppose that for every vector $g\perp{f_0}$, and $\norm{g}\leq1$, there exists an operator $A\in\A$, satisfying the following strict inequality: \begin{equation}\label{e:ESSCON} \RE\seq{A\left(f_0+\tfrac{r}{\sqrt{1-r^2}}g\right), f_0-\tfrac{\sqrt{1-r^2}}{r}g} > \essnorm{\RE{A}}(1 - \norm{g}^2). \end{equation} Then $\A$ contains an operator $A_0$, with an eigenvector in the set \[ \s=\set{f\in\h\,\,|\,\,\,\norm{f_0-f}\leq{r}}, \] and the corresponding eigenvalue $\lambda$ satisfies the condition: $\abs{\RE\lambda}>\essnorm{\RE{A_0}}$. \end{prop} \begin{proof} Introducing the polar hyperplane $\Polar$ as before, observe that by (\ref{e:NUMERICAL}) the condition~(\ref{e:ESSCON}) implies that every vector in $\Polar\cap\s$ satisfies the hypothesis of Lemma~\ref{l:ESS} for some operator $A\in\A$. Consequently, for every vector $f$ in $\Polar\cap\s$ there exists an operator $A\in\A$, together with a (basic) weak neighborhood $\W$ of $f$, and a positive number $\delta$, such that for every $h\in\W\cap\s$: \[ \Delta_A(h) > \essnorm{\RE{A}}\abs{\Delta_I(h)} + \delta. \] By Lemma~\ref{l:APPROX} there exists a positive number $\mu$ such that the operator $I+\eps A$ maps the set $\W\cap\s$ into $\s$ whenever $\eps\in(0,\mu)$. In this way we obtain a weakly open cover of $\Polar\cap\s$ with basic neighborhoods. By the weak-compactness of the set $\Polar\cap\s$, there exists a finite subcover $\W_1,\ldots,\W_n$, together with the operators $A_k$ in $\A$, and positive numbers $\mu_k>0$, such that for $\eps\in(0,\mu_k)$ the operator $I+\eps{A_k}$ maps the set $\W_k\cap\s$ into $\s$, and \begin{equation}\label{e:EINEQ} \Delta_{A_k}(h) > \essnorm{\RE{A_k}} \abs{\Delta_I(h)}, \quad \text{for every $h\in\W_k\cap\s$}. \end{equation} Define the weakly open set $\W_0=\set{f\in\h\,\,|\,\,\,\abs{\seq{f,f_0}-(1-r^2)}>0}$. Associated with the set $\W_0$ is its continuous indicator function $\Gamma_0\colon\h\To\RPlus$: \[ \Gamma_0(f)=\abs{\seq{f-f_0}-(1-r^2)}, \] and the function $\Lambda_0\colon\s\To\s$, defined in Lemma~\ref{l:SCALE}: $\Lambda_0(f)=\big(r^2+\seq{f,f_0}\big)^{-1}f$. Fix a positive number $\eps\in(0,\min\set{\mu_1,\ldots,\mu_n})$, and recall that every basic weak neighborhood $\W_k$ admits a continuous indicator function $\Gamma_k\colon\s\To[0,1]$, defined by~(\ref{e:CIF}). Each point $f\in\s$ lies at least in one neighborhood $\W_k$ $(k=0,\ldots,n)$, therefore the sum $\sum_{j=0}^n\Gamma_j(f)$ is strictly positive for all vectors $f\in\s$. Hence, the functions $\alpha_k\colon\s\To[0,1]$, \[ \alpha_k(f) = \frac{\Gamma_k(f)}{\sum_{j=0}^n \Gamma_j(f)} \quad\quad (k=0,\ldots,n), \] are well defined and weakly continuous on $\s$. Also, $\sum_{k=0}^n\alpha_k(f)=1$ for every $f \in \s$, and $\alpha_k(f)>0$ if and only if $f\in\W_k$. The Lomonosov function $\Lambda\colon\s\To\s$, in the Lomonosov space $\Lom({\A\cup\Lambda_0})$, associated with the set of functions $\A\cup\Lambda_0\subset{C}(\s,\h)$, is defined by \[ \Lambda(f) = \frac{\alpha_0(f)}{r^2+\seq{f,f_0}} f+\sum_{k=1}^n \alpha_k(f) (I+\eps A_k)f. \] Observe, that $\Lambda(f)$ is a convex combination of elements in $\s$, and consequently, $\Lambda$ maps the set $\s$ into itself (weak-continuously). The Schauder-Tychonoff Fixed Point Theorem implies that the Lomonosov function $\Lambda\colon\s\To\s$ has a fixed point $f_1\in\s$. From $\Lambda(f_1)=f_1$, we conclude: \begin{align*} \eps \Big(\sum_{k=1}^n \alpha_k(f_1) A_k\Big)f_1 & = \left(1 - \sum_{k=1}^n \alpha_k(f_1) - \frac{\alpha_0(f_1)}{r^2+\seq{f_1,f_0}} \right)f_1\\ & = \alpha_0(f_1)\left(1 - \frac{1}{r^2+\seq{f_1,f_0}}\right)f_1. \end{align*} Outside the set $\W_1\cup\ldots\cup\W_n$ the function $\Lambda$ equals $\Lambda_0$ and has no fixed points. Consequently, $f_1\in\W_k$ for at least one index $k\in\set{1,\ldots,n}$, and $\sum_{j=1}^n\alpha_j(f_1)>0$. Set \[ \beta_k=\frac{\alpha_k(f_1)}{\sum_{j=1}^n \alpha_j(f_1)}= \frac{\alpha_k(f_1)}{1-\alpha_0(f_1)}, \qquad (k=1,\ldots,n). \] Then $A_0=\sum_{k=1}^n \beta_k A_k$ is an operator in the convex set $\A$. Clearly, $f_1\in\s$ is an eigenvector for $A_0$, corresponding to the eigenvalue $\lambda$: \begin{equation}\label{e:LAMBDA} \lambda = \frac{\alpha_0(f_1)}{\eps\big(1-\alpha_0(f_1)\big)} \left(1 - \frac{1}{r^2+\seq{f_1,f_0}}\right). \end{equation} Recall that by (\ref{e:EINEQ}) the strict inequality $\Delta_{A_k}(f_1) > \essnorm{\RE{A_k}}\abs{\Delta_I(f_1)}$ is satisfied whenever $\alpha_k(f_1)>0$ (or, equivalently: $\beta_k>0$). Therefore, nonnegativity of the coefficients $\beta_k$, and subadditivity of the essential norm, imply: \[ \Delta_{A_0}(f_1) = \sum_{k=1}^n \beta_k \Delta_{A_k}(f_1) > \sum_{k=1}^n \beta_k \essnorm{\RE{A_k}}\abs{\Delta_I(f_1)} \geq \essnorm{\RE{A_0}}\abs{\Delta_I(f_1)}. \] By~(\ref{e:LAMBDA}) the sign of $\IM\lambda$ is the same as the sign of $\IM\seq{f_1,f_0}=\IM\seq{f_1,f_0-f_1}$. Hence, from $A_0f_1=\lambda{f_1}$ and $\Delta_{A_0}(f_1)>\essnorm{\RE{A_0}}\abs{\Delta_I(f_1)}$, we conclude: \begin{align*} \abs{\RE\lambda}\abs{\Delta_I(f_1)} &\geq (\RE\lambda)\Delta_I(f_1) \\ & = \tfrac{1}{r^2(1-r^2)}(\RE\lambda) \RE\seq{f_1, f_0-f_1} \\ &\geq \tfrac{1}{r^2(1-r^2)}\big((\RE\lambda)\RE\seq{f_1, f_0-f_1}- (\IM\lambda)\IM\seq{f_1, f_0-f_1}\big)\\ & = \tfrac{1}{r^2(1-r^2)}\RE\seq{\lambda f_1, f_0-f_1} \\ & = \Delta_{A_0}(f_1) \\ & > \essnorm{\RE{A_0}}\abs{\Delta_I(f_1)}. \end{align*} The strict inequality implies that $\Delta_I(f_1)\neq0$, and consequently $\lambda$ satisfies the required condition: $\abs{\RE\lambda}>\essnorm{\RE{A_0}}$. \end{proof} %%% ---------------------------------------------------------------------- \subsection{Burnside's Theorem Revisited} Lomonosov~\cite{Lom91} established the following extension of Burnside's Theorem to infinite-dimensional Banach spaces: \begin{thm}[V.I.\,Lomonosov, 1991]\label{t:LOMBUR} Suppose $\X$ is a complex Banach space and let $\A$ be a weakly closed proper subalgebra of $\BOP(\X)$, $\A\neq\BOP(\X)$. Then there exists $x\in{\X^{**}}$ and $y\in{\X^*}$, $x\neq0$ and $y\neq0$, such that for every $A\in\A$ \begin{equation}\label{e:LB} \abs{\seq{x,A^*y}}\leq\essnorm{A}. \end{equation} \end{thm} The techniques introduced in the proof of this theorem, based on the argument of the celebrated de~Branges proof of the Stone-Weierstrass Theorem~\cite{dB59}, received further attention in \cite{AAB95,dB93}. Although in the Hilbert space case Theorem~\ref{t:LOMBUR} is equivalent to another theorem, also given in \cite{Lom91}, we take a different point of view and employ Proposition~\ref{p:ESS} to obtain a stronger extension of Burnside's Theorem to infinite-dimensional Hilbert spaces. The condition (\ref{e:LB}) is equivalent to the existence of \emph{unit} elements $x\in{\X^{**}}$ and $y\in{\X^*}$, and a nonnegative constant $C$ (depending on $\A$), such that \begin{equation}\label{e:LBC} \abs{\seq{x,A^*y}}\leq C \essnorm{A}, \qquad \text{ for all $A\in\A$}. \end{equation} In general, the constant $C$ depends on the space $\X$, and the algebra $\A$. It is not clear that on every Banach space there exists an upper bound for $C$, satisfying the condition (\ref{e:LBC}), with respect to all proper weakly closed subalgebras of $\BOP(\X)$. An example of such a space would certainly be of some interest. On the other hand, an affirmative answer to the Transitive Algebra Problem~\cite{RR73} is equivalent to $C=0$. At the moment we can provide no results concerning the estimates for the constant $C$ in any infinite-dimensional Banach space, other than a Hilbert space. The next theorem implies that on a complex Hilbert space the constant $C$ is at most one. \begin{thm}\label{t:UNIFBURN} Suppose $\h$ is a complex Hilbert space and let $\A$ be a weakly closed subalgebra of $\BH$, $\A\neq\BH$. Then there exist nonzero vectors $f,h\in\h$, such that for all $A\in\A$: \begin{equation}\label{e:ReESS} \abs{\RE\seq{A f,h}}\leq\essnorm{\RE A}\seq{f,h}. \end{equation} \end{thm} \begin{proof} Suppose not; then the hypothesis of Proposition~\ref{p:ESS} is satisfied for every unit vector $f_0$ and any positive number $r\in(0,1)$. Consequently, the algebra $\A$ contains an operator $A_0$ with an eigenvalue $\lambda$, satisfying the condition: \[ \abs{\RE\lambda}>\essnorm{\RE{A}}. \] Proposition~\ref{p:ESSPECT} implies that the algebra $\A$ contains a nonzero finite-rank operator. Therefore~\cite[Theorem~8.2]{RR73}, the (transitive) algebra $\A$ is weakly dense in $\BH$, contradicting the assumption $\A\neq\BH$. \end{proof} \begin{rem} Note that (after arbitrarily choosing the unit vector $f_0$ and then letting $r\To0$) the argument in the proof of Theorem~\ref{t:UNIFBURN} shows that, unless $\A$ contains a nonzero finite-rank operator, the set of all vectors $f\in\h$ for which there exists a nonzero vector $g\in\h$ satisfying the condition (\ref{e:ReESS}) is \emph{dense} in $\h$. \end{rem} \begin{cor}\label{c:UNIFBURN} Suppose $\h$ is a complex Hilbert space and let $\A$ be a weakly closed subalgebra of $\BH$, $\A\neq\BH$. Then there exist \emph{unit} vectors $f,h\in\h$, such that for all $A\in\A$: \begin{equation}\label{e:AbsESS} \abs{\seq{A f,h}}\leq\essnorm{A}. \end{equation} \end{cor} \begin{proof} By Theorem~\ref{t:UNIFBURN} there exist unit vectors $f,h\in\h$, such that for every $A\in\A$ \[ \abs{\RE{\seq{A f,h}}}\leq\essnorm{\RE{A}}. \] Set \[ \xi = \left\lbrace \begin{array}{c l} 1 & \text{if $\seq{A{f},h}=0$},\\ \tfrac{\seq{A{f},h}}{\abs{\seq{A{f},h}}} & \text{otherwise}. \end{array} \right. \] Then \[ \abs{\seq{A f,h}} = \abs{\RE\seq{\xi A f,h}} \leq \essnorm{\RE(\xi A)} \leq \essnorm{\xi A} = \essnorm{A}, \] and consequently, the condition (\ref{e:AbsESS}) is weaker than (\ref{e:ReESS}). \end{proof} The following definition yields an alternative formulation of the extended Burnside Theorem. \begin{defn}\label{d:ESSTRANS} A vector $f\in\h$ is called \emph{essentially cyclic} for an algebra $\A\subset\BH$ if for every nonzero vector $g\in\h$, such that $\seq{f,g}\geq0$, there exists an operator $A\in\A$, satisfying the following strict inequality: \[ \RE{\seq{A{f},g}} > \essnorm{\RE{A}} \seq{f,g}. \] We say that a subalgebra $\A$ of $\BH$ is \emph{essentially-transitive} if every nonzero vector is essentially cyclic for $\A$. \end{defn} \begin{rem} Note that our definition of essentially-transitive algebras does not coincide with the definition in~\cite{Lom91}. In view of the discussion preceding Theorem~\ref{t:UNIFBURN}, we required the constant $C$ to be at most one in the definition of essential transitivity, while the definition in~\cite{Lom91} assumes no upper bound on $C$. According to Definition~\ref{d:ESSTRANS}, every essentially cyclic vector $f\in\h$ is also cyclic for $\A$, i.e. the orbit $\set{A{f}\,\,|\,\,\,A\in\A}$ is dense in $\h$. Consequently, an essentially-transitive algebra is also transitive, as defined in~\cite{RR73}. \end{rem} Theorem~\ref{t:UNIFBURN} can be restated as the following solution of the ``Essentially-Transitive Algebra Problem''. \begin{thm}\label{t:ESSBURN} An essentially-transitive algebra of operators acting on a complex Hilbert space $\h$ is weakly dense in $\BH$. \end{thm} \begin{rem} The reader may have noticed that, by Propositions~\ref{p:ESS}~and~\ref{p:ESSPECT}, an essentially-transitive algebra $\A$ acting on a real or complex Hilbert space contains a nonzero finite-rank operator in its norm closure. However, in the case of a real Hilbert space the existence of a nonzero finite-rank operator does not imply that $\A$ is weakly dense in $\BH$. A commutative algebra $\J$, generated by the matrix \[ J= \left[ \begin{array}{rr} 0 & -1 \\ 1 & 0 \end{array} \right], \] is an example of a proper essentially-transitive algebra acting on $\Real^2$. The tensor product $\BH\otimes\J$ is an example of such an algebra acting on $\h\oplus\h$. However, the existence of a nonzero finite-rank operator in the closure of an essentially-transitive algebra, implies the following commutative version of Theorem~\ref{t:ESSBURN}, which holds on real or complex infinite-dimensional Hilbert spaces. \end{rem} \begin{thm}\label{t:COMBURN} A commutative algebra $\A$, of operators acting on a real or complex infinite-dimensional Hilbert space, is \emph{never} essentially-transitive. \end{thm} \begin{proof} By Propositions~\ref{p:ESS}~and~\ref{p:ESSPECT} the (norm) closure of every essentially-transitive algebra contains a nonzero finite-rank operator $T$. Since $T{A}=A{T}$ for every $A\in\A$, it follows that the range of $T$ is a nontrivial (finite-dimensional) invariant subspace for $\A$, contradicting the (essential) transitivity of $\A$. \end{proof} \begin{rem} Note that the applications in this subsection do not use the full power of Proposition~\ref{p:ESS}. In the next section we perform a more detailed analysis of the set of vectors for which the condition (\ref{e:ESSCON}) fails, in a special case when $\A$ is the algebra generated by an essentially self-adjoint operator. \end{rem} %%% ---------------------------------------------------------------------- \section{On Invariant Subspaces of Essentially Self-Adjoint Operators} An application of the main result of the previous section to the algebra generated by an essentially self-adjoint operator $A$ yields the existence of nonzero vectors $x,y\in\h$, such that $\tau(p)=\seq{p(A)x,y}$ is a positive functional on the space of all polynomials on the essential spectrum of $A$. This result immediately implies the existence of real invariant subspaces for essentially self-adjoint operators acting on a complex Hilbert space. Elementary convex analysis techniques, applied to the space of certain vector states, yield the existence of invariant subspaces for essentially self-adjoint operators acting on an infinite-dimensional real Hilbert space. %%% ---------------------------------------------------------------------- \subsection{Introduction} The existence of invariant subspaces for compact perturbations of self-adjoint operators appears to be one of the most difficult questions in the theory of invariant subspaces~\cite{Lom92}. The positive results about the existence of invariant subspaces for the Schatten-class perturbations of self-adjoint operators, acting on a complex Hilbert space, date back to the late 1950's. For the facts concerning such operators see Chapter~6 in~\cite{RR73}, where a brief history of the problem, together with references to the related topics is given. The proofs of those results are based on the concept of the separation of spectra. However, Ljubi\v{c} and Macaev~\cite{LM65} showed that there is no general spectral theory by constructing an example of an operator $A$ such that $\sigma(A|\M)=[0,1]$ whenever $\M$ is a nonzero invariant subspace for $A$. This suggests that different techniques might be needed to establish the existence of invariant subspaces for essentially self-adjoint operators. The fact that the right-hand side of the inequality~(\ref{e:ESSCON}) depends only on the essential norm of the real part of the operator $A$, suggests that Proposition~\ref{p:ESS} might have applications to the invariant subspace problem for compact perturbations of self-adjoint operators. In this section we apply Proposition~\ref{p:ESS} in order to construct positive functionals $\tau(p)=\seq{p(A)x,y}$ on the space of all polynomials restricted to the essential spectrum of $A$. Finally, in the case when the underlying Hilbert space is real, the existence of invariant subspaces for $A$ is established after solving an extreme problem concerning certain convex subspaces of vector states. \begin{defn} Suppose $\h$ is a real or complex Hilbert space. An operator $A\in\BH$ is called \emph{essentially self-adjoint} if $\pi(A)$ is a self-adjoint element in the Calkin algebra $\BH/\KH$, where $\pi\colon\BH\To\BH/\KH$ is the quotient mapping. \end{defn} \begin{rem} Clearly, by definition of the Calkin algebra, $A$ is essentially self-adjoint if and only if $A=S+K$, where $S\in\BH$ is self-adjoint and $K$ is a compact operator. Hence, saying that $A$ is essentially self-adjoint, is the same as saying that $A$ is a compact perturbation of a self-adjoint operator. Note, however, that this is \emph{false} if we replace self-adjoint operators by normal ones. \end{rem} %%% ---------------------------------------------------------------------- \subsection{On Real Invariant Subspaces} Recently Lomonosov~\cite{Lom92} proved that every essentially self-adjoint operator acting on a complex Hilbert space has a nontrivial closed \emph{real} invariant subspace. We give an alternative proof, based on Proposition~\ref{p:ESS}, and thus introduce the idea that will be later generalized in order to yield the existence of proper invariant subspaces for essentially self-adjoint operators acting on a real Hilbert space. Recall that a \emph{real subspace} of a complex Hilbert space $\h$ is a subset that is closed under addition and multiplication by the \emph{real} scalars. A real subspace $\M\subset\h$ is invariant for an operator $A\in\BH$ if and only if $\M$ is invariant under all operators in the \emph{real} algebra generated by $A$, i.e. the algebra of all real polynomials in $A$. \begin{prop}\label{p:RIS} Suppose $\h$ is an infinite-dimensional complex Hilbert space and let $\A$ be a convex set of commuting essentially self-adjoint operators. Then the set of non-cyclic vectors for $\A$ is dense in $\h$. \end{prop} \begin{proof} Suppose not; then there exists a unit vector $f_0$ and a positive number $r\in(0,1)$ such that all vectors in the set \[ \s=\set{f\in\h\,\,|\,\,\,\norm{f_0-f} \leq \tfrac{r}{\sqrt{1-r^2}}}, \] are cyclic for $\A$. In particular, for every vector $g\in\h$ and $\norm{g}\leq1$, there exists an operator $A\in\A$ such that \begin{align*} \RE\seq{A\left(f_0+\tfrac{r}{\sqrt{1-r^2}}g\right), -i\left(f_0-\tfrac{\sqrt{1-r^2}}{r}g\right)} & = \\ \RE\seq{(i{A})\left(f_0+\tfrac{r}{\sqrt{1-r^2}}g\right), \left(f_0-\tfrac{\sqrt{1-r^2}}{r}g\right)} & > 0. \end{align*} Since $A$ is an essentially self-adjoint operator, it follows that \[ \essnorm{\IM{A}}=\essnorm{\RE(i{A})}=0, \] and consequently the convex set $i\A=\set{i{A}\,\,|\,\,\,A\in\A}$, satisfies the hypothesis of Proposition~\ref{p:ESS}. Therefore, there exists an element $A_0\in\A$ ($A_0\neq{z}I$), with an eigenvector $f_1\in\s$. Since the operators in $\A$ commute, $f_1$ cannot be a cyclic vector for $\A$, contradicting the assumption that all vectors in $\s$ are cyclic for $\A$. \end{proof} \begin{cor}[V.I.\,Lomonosov, 1992] Every essentially self--adjoint operator on an infinite-dimensional complex Hilbert space has a nontrivial closed real invariant subspace. \end{cor} \begin{proof} The commutative algebra $\A_{\Real}$ of all real polynomials in $A$ consists of essentially self-adjoint operators whenever $A$ is essentially self-adjoint. By Proposition~\ref{p:RIS} the set of non-cyclic vectors for $\A_{\Real}$ is dense in $\h$. Since for every nonzero vector $f\in\h$ the closure of the orbit $\A_{\Real}f=\set{T{f}\,\,|\,\,\,T\in\A_{\Real}}$ is a real invariant subspace for $A$, it follows that $A$ has a nontrivial closed real invariant subspace. \end{proof} \begin{rem} If $A$ is a self-adjoint operator acting on a complex Hilbert space $\h$ then for every vector $f\in\h$ and every real polynomial $p$ we have: \begin{equation} \label{e:I0} \IM\seq{p(A){f},f}=0. \end{equation} The condition (\ref{e:I0}) in fact characterizes self-adjoint operators on a complex Hilbert space~\cite[p.~103]{KR83}. Roughly speaking, Proposition~\ref{p:RIS} and its corollary establish a similar fact for essentially self-adjoint operators acting on a complex Hilbert space. \end{rem} %%% ---------------------------------------------------------------------- \subsection{The Space of Vector States} In the previous subsection we applied our machinery only to the imaginary part of an essentially self-adjoint operator $A$. An application to the real part yields the existence of ``vector states'' on the space of all polynomials restricted to the essential spectrum of $A$. Before proceeding, we make the following conventions that hold through the rest of this section: As usual, let $\h$ be an infinite-dimensional real or complex Hilbert space. The underlying field of real or complex numbers (respectively) is denoted by $\Field$. Suppose $A\in\BH$ is a fixed essentially self-adjoint operator without non-trivial closed invariant subspaces and let $E$ denote its essential spectrum. Furthermore, we may assume that $\essnorm{A}\leq1$, and consequently, $E\subset[-1,1]$. Let $\A\subset\BH$ be an algebra generated by $A$, i.e. $\A$ is the algebra of all polynomials $p(A)$ with the coefficients in the underlying field $\Field$. The algebra of all polynomials with the coefficients in $\Field$, equipped with the norm \[ \norm{p}_\infty=\max_{t\in{E}} \abs{p(t)}, \] is denoted by $\Poly$. \begin{defn} Let $\EssD\subset\h$ be the set of all nonzero vectors $x\in\h$ for which there exists a nonzero vector $y\in\h$ satisfying the following inequality for every polynomial $p\in\Poly$: \begin{equation} \label{e:PF1} \RE\seq{p(A)x,y} \leq \norm{\RE p}_\infty\seq{x,y}. \end{equation} \end{defn} \begin{lem}\label{l:PF1} The set $\EssD$ is dense in $\h$. \end{lem} \begin{proof} Since the operator $A$ has no invariant subspaces the condition of Proposition~\ref{p:ESS} is never satisfied for the algebra $\A$. More precisely, for every unit vector $f_0\in\h$ and any positive number $r\in(0,1)$ there exists a vector $g\perp{f_0}$ such that for every polynomial $p\in\Poly$ \[ \RE\seq{p(A)\left(f_0+\tfrac{r}{\sqrt{1-r^2}}g\right), f_0-\tfrac{\sqrt{1-r^2}}{r}g} \leq \essnorm{\RE{p(A)}}(1 - \norm{g}^2). \] Clearly, for every polynomial $p\in\Poly$ we have: \[ \essnorm{\RE p(A)} = \essnorm{(\RE p)(A)} = \norm{\RE p}_\infty. \] The vectors $x=f_0+\tfrac{r}{\sqrt{1-r^2}}g$ and $y=f_0-\tfrac{\sqrt{1-r^2}}{r}g$ satisfy the inequality (\ref{e:PF1}). Letting $r\to0$ implies the required density of $\EssD$. \end{proof} \begin{lem} \label{l:PF2} For fixed vectors $x,y\in\h$, a linear functional $\tau\colon\Poly\To\Field$ is defined by \[ \tau(p) = \seq{p(A)x,y}. \] Then $\tau$ is a bounded positive functional on the space $\Poly$ if and only if the following inequality is satisfied for every polynomial $p\in\Poly$: \begin{equation}\label{e:PF1A} \RE\seq{p(A)x,y} \leq \norm{\RE p}_\infty\seq{x,y}. \end{equation} \end{lem} \begin{proof} Suppose that $\tau$ is a positive functional on $\Poly$. Then $\RE \seq{p(A)x,y} = \seq{(\RE p)(A)x,y}$. Since $\norm{\RE p}_\infty - \RE p$ is a positive polynomial on $E$, we have: \[ \tau(\norm{\RE p}_\infty - \RE p) = \seq{(\norm{\RE p}_\infty - \RE p)(A)x, y} \geq 0, \] or, equivalently: \[ \RE\seq{p(A)x,y} \leq \norm{\RE p}_\infty\seq{x,y}. \] Conversely, suppose $\tau$ is not a bounded positive functional on $\Poly$. Then either there exists a real polynomial $p$ such that $\IM\seq{p(A)x,y}\neq0$, or $\seq{p(A)x,y}<0$ for some positive polynomial $p\in\Poly$. After replacing $p$ by $\pm{i}{p}$ it is easy to see that $\IM\seq{p(A)x,y}\neq0$ contradicts (\ref{e:PF1A}). Similarly, for a positive polynomial $p$ we have: \[ \norm{\norm{p}_\infty-p}_\infty \leq \norm{p}_\infty. \] Therefore $\seq{p(A)x,y}<0$ and $\seq{x,y}\geq0$ imply: \[ \seq{ \big(\norm{p}_\infty-p(A)\big)x,y} > \norm{p}_\infty\seq{x,y} \geq \norm{\norm{p}_\infty-p}_\infty\seq{x,y}, \] contradicting (\ref{e:PF1A}). Finally, in the case when $\seq{x,y}<0$ the inequality (\ref{e:PF1A}) fails for the constant polynomial $p\equiv-1$. \end{proof} \begin{defn} The set of all bounded positive linear functionals on $\Poly$ is denoted by $\States$. For each vector $x\in\h$ define the set \[ \States_x = \set{y\in\h\,\,|\,\,\, \tau(p) = \seq{p(A)x,y} \in \States} . \] \end{defn} \begin{rem} Observe that a linear functional $\tau\colon\Poly\to\Field$ is in $\States$ if and only $\norm{\tau}=\tau(1)$. This observation follows from the similar (well-known) characterization of positive functionals on C*-algebras~\cite{KR83}. \end{rem} \begin{lem} \label{l:PF3} For every vector $x\in\h$, $\States_x$ is a closed convex subset of $\h$. \end{lem} \begin{proof} Convexity of the set $\States_x$ is obvious. It remains to prove that the complement of $\States_x$ is an open subset of $\h$. If $y\not\in\States_x$ then there exists a positive polynomial $p\in\Poly$ such that $\seq{p(A)x, y} \not\geq 0$. In that case there exists a weak neighborhood $\W$ of $y$ such that $\seq{p(A)x,z}\not\geq0$ for every $z\in\W$. Consequently, the complement of the set $\States_x$ is a (weakly) open subset of $\h$. \end{proof} \begin{defn} A positive functional $\tau\in\States$ is called a \emph{state} if $\norm{\tau}=1$, or, equivalently: $\tau(1)=1$. The space of all states on $\Poly$ is denoted by $\States'$. Similarly, for every vector $x\in\h$ the set $\States'_x$ is defined by \[ \States'_x = \set{y\in\h\,\,|\,\,\, \tau(p) = \seq{p(A)x,y} \in \States'}. \] \end{defn} \begin{rem} From Lemma~\ref{l:PF1} and Lemma~\ref{l:PF2} it follows that the set $\EssD$ of all vectors $x\in\h$ for which the set $\States_x$ contains a nonzero vector is dense in $\h$. If $x$ and $y$ are nonzero vectors and $y\in\States_x$ then $\seq{x,y}\geq0$. However, since a positive functional always attains its norm on the identity function, the equality $\seq{x,y}=0$ implies that $\tau(p)=\seq{p(A)x,y}=0$ for every polynomial $p\in\Poly$, contradicting the fact that the operator $A$ has no invariant subspaces. Therefore, the set $\States'_x$ is nonempty for every vector $x$ in a dense set $\EssD\subset\h$. In fact, for every vector $x\in\EssD$ the set $\States'_x$ is the intersection of the cone $\States_x$ and the hyperplane $\M_x=\set{y\in\h\,\,|\,\,\,\seq{y,x}=1}$. Note also, that for nonzero vectors $x\in\EssD$ and $y\in\States_x$, we have: $\seq{x,y}^{-1}y\in\States'_x$. \end{rem} By Lemma~\ref{l:PF3} the set $\States_x$ is a weakly closed convex subset of $\h$. We show that the set $\States'_x$ has no extreme points. \begin{lem} \label{l:EXTR} For every vector $x\in\h$ the set $\States'_x$ has no extreme points. \end{lem} \begin{proof} Suppose $y_0$ is an extreme point in $\States'_x$. By definition of the set $\States'_x$, the functional $\tau(p)=\seq{p(A)x,y_0}$ is a state on $\Poly$. Hence, \[ \omega(p)=\tau((1-t)p(t))=\seq{p(A)x, (1-A^*)y_0} \] is a positive functional on $\Poly$. Consequently, \[ y_1 = \seq{(1-A)x, y_0}^{-1}(1-A^*)y_0 \in \States'_x. \] Similarly, \[ y_2 = \seq{(1+A)x, y_0}^{-1}(1+A^*)y_0 \in \States'_x. \] From \[ y_0 = \frac{\seq{(1-A)x, y_0}}{2} y_1 + \frac{\seq{(1+A)x, y_0}}{2} y_2, \] we conclude that $y_0=y_1=y_2$. Therefore, $(1-A^*)y_0=\seq{(1-A)x,y}y_0$ implies that $y_0$ is an eigenvector for $A^*$, contradicting the nonexistence of invariant subspaces for the operator $A$. \end{proof} \begin{cor} \label{c:UNBOUND} For every vector $x\in\h$ the set $\States'_x$ is either empty or unbounded. \end{cor} \begin{proof} By the Krein-Milman Theorem the set $\States'_x$ cannot be weakly compact due to the lack of extreme points. \end{proof} Although the set $\States'_x$ is unbounded for every vector $x\in\EssD$, the following lemma shows that it contains no line segments of infinite length. In particular, $\States'_x$ is a \emph{proper} subset of the hyperplane \[ \M_x=\set{y\in\h\,\,|\,\,\,\seq{y,x}=1}. \] \begin{lem} \label{l:FL} Every line segment in $\States'_x$ has a finite length. \end{lem} \begin{proof} Suppose the set $\States'_x$ contains a line segment of infinite length. Then there exists a vector $y\in\States'_x$, and a unit vector $u\perp{x}$ such that $y+\lambda u\in\States'_x$ for every $\lambda\geq0$. For every power $k\geq0$, and every vector $z\in\States'_x$, we have: $\abs{\seq{A^k x,z}}\leq1$. Applying this inequality to a vector $y+\lambda{u}$, and letting $\lambda\to\infty$, implies that $\seq{A^k x,u}=0$, contradicting the fact that $x$ is a cyclic vector for $A$. \end{proof} %%% ---------------------------------------------------------------------- \subsection{Invariant Subspaces on a Real Hilbert Space} In this subsection we use vector states in order to establish the existence of invariant subspaces for essentially self-adjoint operators acting on an infinite-dimensional real Hilbert space. The invariant subspace problem for essentially self-adjoint operator is translated into an extreme problem and the solution is obtained upon differentiating certain functions at their extreme. Once again we employ the differentiability of the Hilbert norm. We start with the following lemma. \begin{lem}\label{l:DIFF} Suppose $x$ and $y$ are any vectors in $\h$ such that $\RE\seq{x,y}=1$. Fix a nonzero operator $T\in\BH$ and let $a=(\norm{T}\norm{x}\norm{y})^{-1}$. Then for every vector $z\in\h$ the function $\psi(\lambda)\colon(-a,a)\To\RPlus$, defined by \[ \psi(\lambda) = \norm{ \big(\RE\seq{(1+\lambda T)y, x}\big)^{-1} (1+\lambda T)y - z}^2, \] is differentiable on $(-a,a)$. Furthermore, if $\psi'$ denotes its derivative then \[ \psi'(0) = 2\RE\seq{T y, y - z - (\norm{y}^2-\RE\seq{y,z}) x}. \] \end{lem} \begin{proof} Since for $\lambda\in(-a,a)$ we have $\RE\seq{(1+\lambda T)y, x}>0$, it follows that the function $\psi$ is well defined on $(-a,a)$. In order to compute its derivative $\psi'(0)$ first apply the polar identity to $\psi$ and then use the product and chain rules for differentiation. A straightforward calculation yields the required formula. \end{proof} \begin{defn} For every vector $x\in\EssD$, define $P_x\colon\h\To\States'_x$ to be the projection to the set $\States'_x$, \,\,i.e. for every $z\in\h$ \[ \norm{P_x z - z} = \inf_{y\in\States'_x} \norm{y-z}. \] \end{defn} \begin{rem} Since for $x\in\EssD$ the set $\States'_x$ is nonempty, closed, and convex it follows that the projection $P_x$ is well defined on the whole space $\h$. \end{rem} \begin{lem}\label{l:REIS} If $x\in\EssD$ then for every vector $z\in\h$ and every power $k=0,1,\ldots$, the following condition is satisfied: \[ \RE\seq{A^k\big((\norm{P_x z}^2 - \RE\seq{P_x z,z})x + (I-P_x)z\big), P_x z} = 0. \] \end{lem} \begin{proof} Let $T=A^{*k}$, and fix a vector $y\in\States'_x$. The function $\Phi(\lambda)\colon(-1,1)\To\States'_x$ is defined by \[ \Phi(\lambda) = \seq{(1+\lambda T)y, x}^{-1} (1+\lambda T)y. \] The same argument as in the proof of Lemma~\ref{l:EXTR} shows that $\Phi$ is well defined and $\Phi(\lambda)\in\States'_x$ for every $\lambda\in(-1,1)$. Choose any vector $z\in\h$ and consider the function $\psi(\lambda)\colon(-1,1)\To\RPlus$, defined by \[ \psi(\lambda)=\norm{\Phi(\lambda) - z}^2. \] By Lemma~\ref{l:DIFF} the function $\psi$ is differentiable, and \[ \psi'(0) = 2\RE\seq{T y, y - z - (\norm{y}^2-\RE\seq{y,z}) x}. \] By definition of the projection $P_x$ the function $\psi$ attains its global minimum at the point $\lambda=0$ whenever $y=P_x z$. Consequently, $\psi'(0)=0$ for $y=P_x z$, which completes the proof. \end{proof} A remarkable fact is that Lemma~\ref{l:REIS} holds on both real and complex infinite-dimensional Hilbert spaces. It is now easy to establish the existence of proper invariant subspaces for essentially self-adjoint operators acting on a real Hilbert space. \begin{thm}\label{t:ISR} Every essentially self-adjoint operator acting on a real infinite-dimensional Hilbert space $\h$ has a nontrivial closed invariant subspace. \end{thm} \begin{proof} Suppose $A$ is an essentially self-adjoint operator acting on a real infinite-dimensional Hilbert space $\h$. We may assume that $\essnorm{A}\leq1$. If the operator $A$ has no nontrivial invariant subspaces then we can apply Lemma~\ref{l:REIS} and Lemma~\ref{l:FL}. We will show that this contradicts the non-existence of invariant subspaces for $A$. On a real Hilbert space Lemma~\ref{l:REIS} implies that for every $k\geq0$, and for any $z\in\h$: \begin{align*} \seq{A^k\big((\norm{P_x z}^2 - \seq{P_x z,z})x + (I-P_x)z\big),P_x z} & = \\ \RE\seq{A^k\big((\norm{P_x z}^2 - \RE\seq{P_x z,z})x + (I-P_x)z\big),P_x z} & = 0 . \end{align*} Since $P_x z\neq0$ it follows that \[ y_z = \big(\norm{P_x z}^2 - \seq{P_x z,z}\big)x + (I-P_x)z \] is a non-cyclic vector for $\A$ whenever $x\in\EssD$. The proof is therefore completed if we show that $y_z\neq0$ for a suitable choice of the vector $z\in\h$. Recall that the set $\States'_x$ lies in the hyperplane $\M_x=\set{y\in\h\,\,|\,\,\,\seq{y,x}=1}$. By definition of the projection $P_x$, the vector $y_z=0$ for $z\in\M_x$ if and only if $z\in\States'_x$. Lemma~\ref{l:FL} implies that $\States'_x$ is a proper subset of the hyperplane $\M_x$ and thus completes the proof. \end{proof} \begin{rem} Theorem~\ref{t:ISR} yields the existence of invariant subspaces for an essentially self-adjoint operator $A$ acting on a complex Hilbert space, whenever the operator $A$ has an essentially self-adjoint matrix representation with real entries. Although considerable efforts have been made to reduce the general complex case to the real one, so far all such attempts have been unsuccessful. \end{rem} \begin{rem} Recall that a vector $y\in\States'_x$ is called a \emph{support point} of the closed convex set $\States'_x$ if $y=P_x{z}$ for some vector $z\not\in\States'_x$. While a proper closed convex subset may have no extreme points it has ``plenty'' of support points. The proof of Lemma~\ref{l:EXTR} implies that the extreme points of the set $\States'_x$ (if they exist) correspond to the eigenvectors of the operator $A^*$. Furthermore, the main result of this section states that every support point of $\States'_x$ is a non-cyclic vector for the real algebra generated by the operator $A^*$. \end{rem} \emph{We suggest that further research in this direction is likely going to reveal additional properties of essentially self-adjoint operators and thus contribute to our understanding of how such operators act on the underlying Hilbert space in terms of invariant subspaces.} % ------------------------------------------------------------------------ \subsection*{Acknowledgment} The results of this paper were obtained during my Ph.D. studies at Dalhousie University and are also contained in my thesis~\cite{Sim94} with the same title. I would like to express deep gratitude to my supervisor and friend Heydar Radjavi whose guidance and support were crucial for the successful completion of this project. % ------------------------------------------------------------------------ %Included for Gather Purpose only: %input "Xbib.bib" \bibliographystyle{amsplain} \bibliography{xbib} \end{document} % ------------------------------------------------------------------------