What are Feedforward Networks & why are they important for neural networks

A machine learning model is a mathematical representation of a real-world process. It is designed to learn from and make predictions or decisions without being explicitly programmed to perform the task. ML models can be trained using a variety of algorithms and techniques to optimize their performance on a given task. Examples include linear regression, decision trees, and neural networks.

Talking about neural networks, they are considered today as a magic box capable of achieving mindblowing things. But what is a neural network?

A neural network is a machine-learning model inspired by the structure and function of the human brain. It can be used for various tasks, such as image recognition, natural language processing, and predictive modelling. But moreover, it can be used to generate images (DALL.E), create chatbots (chatGPT), self-driving cars and much more.

Neural networks are particularly well suited for tasks that involve large amounts of complex, unstructured data, such as images or text. They can also identify patterns and relationships in data that are difficult or impossible for humans to detect.

But how are they started, and how are they related to feedforward networks?

Let’s first define feedforward networks.

A feedforward network is a type of neural network in which the information flows through the network in only one direction, from the input nodes, through the hidden layers, and to the output nodes. If you are familiar with stacks, it looks like this:

The input data is passed through a series of layers, each of which performs a mathematical transformation on the data, and the output of one layer is used as the input to the next layer. This process continues until the output is produced.

The most important factor here is that there are no loops or cycles in the network, hence the name feedforward. The feedforward network architecture is widely used in many applications, such as image classification, speech recognition, and natural language processing.

Mathematically it is just a composition of functions, and you can see that stack as being this final function :

with f¹ being the function 1 on the stack represented on the image up. Feedforward networks are the foundations of neural networks mimicking the layers architecture of the neural networks, with each layer being a function and the complete neural network being finally a composition of multiple functions.

Feedforward networks are used as building blocks for many neural network architectures. Here are some examples of neural network architectures that use feedforward networks:

1. Multi-layer Perceptron (MLP): A basic feedforward neural network that consists of an input layer, one or more hidden layers, and an output layer.
2. Convolutional Neural Network (CNN): A feedforward neural network commonly used in image and video processing tasks. It uses convolutional layers to automatically and adaptively learn spatial hierarchies of features from input data.
3. Autoencoder: A feedforward neural network that is trained to reconstruct its input. It consists of an encoder and a decoder. The encoder maps the input to a lower-dimensional representation, and the decoder maps the lower-dimensional representation back to the original input.
4. Radial basis function network (RBFN): A feedforward neural network that uses radial basis functions as activation functions in the hidden layer.
5. Time delay neural network (TDNN): A feedforward neural network that processes sequential data. It uses a time delay mechanism in the input layer to capture temporal dependencies in the data.

These are just a few examples of feedforward neural networks, and many other architectures can be designed using feedforward networks as the building block.

But some categories of neural networks are built differently, like Recurrent neural networks that create a chain by feeding the last output to the initial input. And those networks are more interesting for natural language processing.