Nonwandering sets and Special $\alpha$-limit sets
of Monotone maps on Regular Curves

In the last two decades, a wide literature on the dynamical properties of maps on some one-dimensional continua has developed. Examples of continua become increasingly studied include, graphs, dendrites and local dendrites (see for instance [1], [2], [3], [30]).

Recently a large class of continua called regular curves has given a special attention (see for example, [5], [12], [13], [14], [20], [21], [28], [29], [31]). These form a large class of continua which includes local dendrites. The Sierpiński triangle is a well known example of a regular curve which is not a local dendrite. Regular curves appear in continuum theory and also in other branches of Mathematics such as complex dynamics; for instance, the Sierpiński triangle can be realized as the Julia set of the complex polynomial $p(z)=z^{2}+2$ (see [9]). Recall that the Julia set of $p$ is $J(p)=\{z\in\mathbb{C}:(p^{n}(z))_{n\geq 1}\textrm{ is bounded}\}$. Seidler [31] proved that every homeomorphism of a regular curve has zero topological entropy (later, this result was extended by Kato in [20] to monotone maps). In [28], [29], Naghmouchi proved that any $\omega$-limit set (resp. $\alpha$-limit set) of a homeomorphism $f$ on a regular curve is a minimal set. Moreover he established the equality between the set of nonwandering points and the set of almost periodic points.  In [12], the first author gave a full characterization of minimal sets for homeomorphisms without periodic points on regular curves. In [15] it was shown that the set of periodic points is either empty or dense in the set of non-wandering points, for homeomorphisms on regular curves. In the present paper, we deal with several questions/problems.

First, we address the question of the equality between the set of nonwandering points and the set of almost periodic points. This was proved in two cases:

$-$ for homeomorphisms on regular curves (Naghmouchi [29])

$-$ for monotone maps on local dendrites (Abdelli et al. [3])

In Theorem 3.1, we prove a more general result by showing that this is true for monotone maps on regular curves. Nevertheless, the later result is false in general for monotone maps on dendroids as it is shown in Example 3.5. Moreover, we show in Theorem 3.3 that the set of nonwandering points coincides with the set of points belonging to their special $\alpha$-limit sets (Proposition 6.8).

Second, beside the usual limits sets $\omega$-limit and $\alpha$-limit, we are interested in the study of another kind of limit sets, called special $\alpha$-limit set (see Section 6). We ask the question whether every special $\alpha$-limit set is a minimal set? We show that for a monotone map $f$ on a regular curve, every special $\alpha$-limit set $s\alpha_{f}(x)$ of a non periodic point $x$ is a minimal set. However, we built an example of a monotone map $f$ on an infinite star for which $s\alpha_{f}(x)$ is not a minimal set for some periodic point $x$. Moreover we show that the inclusion $s\alpha_{f}(x)\subset\alpha_{f}(x)$ is strict. In addition, we prove that $\textrm{SA}(f)=\textrm{R}(f)$, where $\textrm{SA}(f)$ respectively $\textrm{R}(f)$ denotes the union of all special $\alpha$-limit sets and the set of recurrent points of $f$. Notice that it is shown recently that, for mixing graph maps $f:G\to G$, every special $s\alpha$-limit set is the $\omega$-limit set of some point from $G$ and moreover, for graph maps $f:G\to G$ with zero topological entropy, every special $s\alpha$-limit set is a minimal set (cf. [24, Theorem 3.8]). On the other hand, Kolyada et al. showed in ([22, Theorem 3.3 and Corollary 3.11]), that $s\alpha_{f}(x)$ is closed whenever $f$ is either an interval map for which the set of all periodic points is closed or $f$ is transitive. Recently, Hantáková and Roth [16] showed that a special $\alpha$-limit set is closed for a piecewise monotone interval map. It is previously known that $s\alpha_{f}(x)$ is closed for homeomorphisms on any compact metric space. In Corollary 6.7, we extend the later result to monotone maps on regular curves by showing that for any $x\in X$, $s\alpha_{f}(x)$ is closed. In fact we show more precisely that for every $x\in X_{\infty}$, $\alpha_{f}(x)\cap\Omega(f)=s\alpha_{f}(x)$ (cf. Theorem 6.2).

Let $X$ be a compact metric space with metric $d$ and let $f:X\to X$ be a continuous map. The pair $(X,f)$ is called a dynamical system. Let $\mathbb{Z},\ \mathbb{Z}_{+}$ and $\mathbb{N}$ be the sets of integers, non-negative integers and positive integers, respectively. For $n\in\mathbb{Z_{+}}$, denote by $f^{n}$ the $n$-th iterate of $f$; that is, $f^{0}=\textrm{identity}$ and $f^{n}=f\circ f^{n-1}$ if $n\in\mathbb{N}$. For any $x\in X$, the subset $\textrm{Orb}_{f}(x)=\{f^{n}(x):n\in\mathbb{Z}_{+}\}$ is called the orbit of $x$ (under $f$). A subset $A\subset X$ is called $f-$invariant (resp. strongly $f-$invariant) if $f(A)\subset A$ (resp., $f(A)=A$); it is further called a minimal set (under $f$) if it is closed, non-empty and does not contain any $f-$invariant, closed proper non-empty subset of $X$. We define the $\omega$-limit set of a point $x\in X$ to be the set:

The set $\alpha_{f}(x)=\displaystyle\bigcap_{k\geq 0}\displaystyle\overline{\bigcup_{n\geq k}f^{-n}(x)}$ is called the $\alpha$-limit set of $x$. Equivalently a point $y\in\alpha_{f}(x)$ if and only if there exist an increasing sequence of positive integers $(n_{k})_{k\in\mathbb{N}}$ and a sequence of points $(x_{k})_{k\geq 0}$ such that $f^{n_{k}}(x_{k})=x$ and $\displaystyle\lim_{k\rightarrow+\infty}x_{k}=y.$ It is much harder to deal with $\alpha$-limit sets since there are many choices for points in a backward orbit. When $f$ is a homeomorphism, $\alpha_{f}(x)=\omega_{f^{-1}}(x)$.

Balibrea et al. [6] considered exactly one branch of the backward orbit as follows.

It is clear that $\alpha_{f}((x_{n})_{n})\subset\alpha_{f}(x)$. The inclusion can be strict; even for onto monotone maps on regular curves (see Example 6.10). Notice that we have the following equivalence:

(i) $\alpha_{f}(x)\neq\emptyset$, (ii) $x\in X_{\infty}$. In particular, if $f$ is onto, then $\alpha_{f}(x)\neq\emptyset$ for every $x\in X$.

A point $x\in X$ is called:

$-$ Periodic of period $n\in\mathbb{N}$ if $f^{n}(x)=x$ and $f^{i}(x)\neq x$ for $1\leq i\leq n-1$; if $n=1$, $x$ is called a fixed point of $f$ i.e. $f(x)=x$;

$-$ Almost periodic if for any neighborhood $U$ of $x$ there is $N\in\mathbb{N}$ such that $\{f^{i+k}(x):0\leq i\leq N\}\cap U\neq\emptyset$, for all $k\in\mathbb{N}$.

$-$Recurrent if $x\in\omega_{f}(x)$.

$-$ Nonwandering if for any neighborhood $U$ of $x$ there is $n\in\mathbb{N}$ such that $f^{n}(U)\cap U\neq\emptyset$.

We denote by P$(f)$, AP$(f)$, R$(f)$, $\Omega(f)$ and $\Lambda(f)$ the sets of periodic points, almost periodic points, recurrent points, nonwandering points and the union of all $\omega$-limit sets of $f$, respectively. Define the space

$X_{\infty}=\displaystyle\bigcap_{n\in\mathbb{N}}f^{n}(X)$. From the definition, we have the following inclusions:

In the definitions below, we use the terminology from Nadler [27] and Kuratowski [23].

When $f$ is closed and onto, Definition 2.3 is equivalent to that the preimage of any point by $f$ is connected (cf. [23, page 131]). In particular, this holds if $X$ is compact and $f$ is onto. Notice that $f^{n}$ is monotone for every $n\in\mathbb{N}$ when $f$ itself is monotone.

A continuum is a compact connected metric space. An arc $I$ (resp. a circle) is any space homeomorphic to the compact interval $[0,1]$ (resp. to the unit circle $\mathbb{S}^{1}=\{z\in\mathbb{C}:\ |z|=1\}$). A space is called degenerate if it is a single point, otherwise; it is non-degenerate. By a graph $G$, we mean a continuum which can be written as the union of finitely many arcs such that any two of them are either disjoint or intersect only in one or both of their endpoints.

A dendrite is a locally connected continuum which contains no circle. Every subcontinuum of a dendrite is a dendrite ([27, Theorem 10.10]) and every connected subset of $D$ is arcwise connected. A local dendrite is a continuum every point of which has a dendrite neighborhood. By ([23, Theorem 4, page 303]), a local dendrite is a locally connected continuum containing only a finite number of circles. As a consequence every subcontinuum of a local dendrite is a local dendrite ([23, Theorems 1 and 4, page 303]). Every graph and every dendrite is a local dendrite.

A regular curve is a continuum $X$ with the property that for each point $x\in X$ and each open neighborhood $V$ of $x$ in $X$, there exists an open neighborhood $U$ of $x$ in $V$ such that the boundary set $\partial U$ of $U$ is finite. Each regular curve is a $1$-dimensional locally connected continuum. It follows that each regular curve is locally arcwise connected (see [23] and [27], for more details). In particular every local dendrite is a regular curve (cf. [23, page 303]).

A continuum $X$ is said to be finitely suslinean continuum provided that each infinite family of pairwise disjoint continua is null (i.e. for any disjoint open sets $U,V$, only a finite number of elements of the family meet both $U$ and $V$). Note that each regular curve is finitely suslinean (see [25]). A continuum is called hereditarily locally connected continuum, written hlc, provided that every subcontinuum of $X$ is locally connected. In particular, finitely suslinean continua and (hence regular curves) are hlc (see [23]). A continuum $X$ is said to be rational curve provided that each point $x$ of $X$ and each open neighborhood $V$ of $x$ in $X$, there exists an open neighborhood $U$ of $x$ in $V$ such that the boundary set $\partial U$ of $U$ is at most countable. Clearly, regular curve are rational.

Let $X$ be a compact metric space. We denote by $2^{X}$ (resp. $C(X)$) the set of all non-empty compact subsets (resp. compact connected subsets) of $X$. The Hausdorff metric $d_{H}$ on $2^{X}$ (respectively $C(X)$) is defined as follows: $d_{H}(A,B)=\max\Big{(}\sup_{a\in A}d(a,B),~{}\sup_{b\in B}d(b,A)\Big{)}$, where $A,B\in 2^{X}$ (resp. $C(X)$). For $x\in X$ and $M\in 2^{X}$, $d(x,M)=\inf_{y\in M}d(x,y)$. With this distance, $(C(X),d_{H})$ and $(2^{X},d_{H})$ are compact metric spaces. Moreover if $X$ is a continuum, then so are $2^{X}$ and $C(X)$ (see [27], for more details). Let $f:X\to X$ be a continuous map of $X$. We denote by $2^{f}:2^{X}\to 2^{X},A\to f(A)$, called the induced map. Then $2^{f}$ is also a continuous self mapping of $(2^{X},d_{H})$ (cf. [27]). For a subset $A$ of $X$, we denote by diam$(A)=\sup_{x,y\in A}d(x,y)$ and $\textrm{card}(A)$ the cardinality of $A$. For any $A\in C(X)$, we denote by

A family $(A_{i})_{i\in I}$ of subsets of $X$ is called a null family if for any infinite sequence $(i_{n})_{n\geq 0}$ of $I,\;\displaystyle\lim_{n\to+\infty}\textrm{diam}(A_{i_{n}})=0$. It is well known that each pairwise disjoint family of subcontinua of a regular curve is null (see [25]).

We recall some results which are needed for the sequel.

Notice that the term weak incompressibility seems to have appeared first in [7].

Let $f$ be a monotone map on a regular curve $X$. If $M$ is an infinite minimal set of $f$, we call $\mathcal{B}(M)=\{x\in X:\omega_{f}(x)=M\}$ the basin of attraction of $M$.

The aim of this section is to prove the following theorem which extends [3, Theorem 1.1] and [29, Theorem 2.1].

In [11], Coven and Nitecki have shown that for continuous maps $f$ of the closed interval $[0,1]$, $\Omega(f)=\{x\in[0,1]:x\in\alpha_{f}(x)\}.$ Later, it is extended to graph maps in [26, Corollary 1]. However, for dendrite maps, it does not holds (see [32]). The following theorem extend the above result to monotone maps on regular curves.

In the following example, we show that Theorem 3.1 cannot be extended to rational curves; we construct a self monotone map $f$ on a rational curve $D$ (it is in fact a dendroid) such that the inclusion is strict: $\textrm{R}(f)\subsetneq\Omega(f)$.

Notice also that Theorem 3.1 does not hold even for continuous maps on intervals; for instance; we may extend the map $g:L\to L$ (defined in Example 3.5) into a continuous map $h$ of $[0,1]$ so that $T_{1}\in\Lambda(h)\subset\Omega(h)$ but $T_{1}\notin\textrm{R}(h)$.

The main result of this section is to prove the following theorem.