Plotting functions with LaTeX & Tikz

I’ve been working on some notes for my students on Neural Networks. I am using Pandoc to write the text, and convert it back to LaTeX for PDF generation, and to HTML, for making the documents available online. In a future post I might talk about Pandoc, but for now, I want to share something I found on TeX Stack Exchange to plot functions. To make it clear how to use, I am presenting four simple examples: four common activation functions used in Neural Networks.

To start with, the well known sigmoid function: \[a = \frac{1}{1+e^{-z}}\]:

This can be achieved with the following tikzpicture environment:

\begin{tikzpicture}
\begin{axis}[
    axis lines=middle,
    xmax=10,
    xmin=-10,
    ymin=-0.05,
    ymax=1.05,
    xlabel={$x$},
    ylabel={$y$}
]
\addplot [domain=-9.5:9.5, samples=100,
          thick, blue] {1/(1+exp(-x)};
\end{axis}
\end{tikzpicture}

Now, for the tanh, defined as \[a=\frac{e^z – e^{-z}}{e^z + e^{-z}}\], you get:

With the following code:

\begin{tikzpicture}
\begin{axis}[
    axis lines=middle,
    xmax=10,
    xmin=-10,
    ymin=-1.05,
    ymax=1.05,
    xlabel={$x$},
    ylabel={$y$}]
\addplot [domain=-9.5:9.5, samples=100,
     thick, blue] {(exp(x) - exp(-x))/(exp(x) + exp(-x))};
\end{axis}
\end{tikzpicture}

I was already loving it, but it was awesome when I tested the ReLU math, and it worked as expected: \[a=\max(0, z)\]

This one can be obtained using the simple…

\begin{tikzpicture}
\begin{axis}[
    axis lines=middle,
    xmax=6,
    xmin=-6,
    ymin=-0.05,
    ymax=5.05,
    xlabel={$x$},
    ylabel={$y$}]
\addplot [domain=-5.5:5.5, samples=100, thick, blue] {max(0, x)};
\end{axis}
\end{tikzpicture}

And now, although not being extremely interesting, the Leaky ReLU (well, not implemented exactly as the definition, just to get it a little more easy to read): \[a=max(0.001 \times z, z)\]

With the cheating…

\begin{tikzpicture}
\begin{axis}[
    axis lines=middle,
    xmax=6,
    xmin=-6,
    ymin=-1.05,
    ymax=5.05,
    xlabel={$x$},
    ylabel={$y$}]
\addplot [domain=-5.5:5.5, samples=100,
          thick, blue] {max(0.1 * x, x)};
\end{axis}
\end{tikzpicture}

And to conclude, and just for reference, do not forget to load the required packages:

\usepackage{tikz}
\usepackage{pgfplots}