Introduction
Deep neural networks are used in modern representation learning systems for vision to integrate extremely high-dimensional pictures into low-dimensional subspaces. These representations, known as embeddings, provide a variety of functions and can be used for downstream tasks including recognition, retrieval, and detection. Images are continuously added in a widely distributed manner to huge systems in the wild. It can be difficult to update the models in these kinds of systems. the new model architecture and training task may be completely disjoint from that of the old model, and therefore its embed-dings are incompatible with downstream tasks reliant on the old embeddings.
Recalculating embeddings (backfilling) for each image is the simplistic solution to feature incompatibility, but it is expensive. Backward Compatible Training (BCT), an approach for the new model to make sure its embeddings are compatible with those of the old model, was introduced by [43] to eliminate the backfilling expense. BCT is able to sustain retrieval accuracy in the absence of backfilling. We demonstrate, however, that for the new model, BCT struggles to produce the same performance improvement as independent training.
The notion of forward-compatible training comes at this point in time. This idea is derived from software engineering, where software is created with the intention of being updated and is therefore designed to be simple to update. A similar strategy is adopted for neural networks. Knowing that there will be an update in the future, we prepare the old model for training so that it will be compatible with some unidentified successor model.
In this article, we will look into the details of forward-compatible training for representation learning to mitigate the drawbacks of backwards-compatible training.
Table of contents:
- What is forward compatible training (FCT)?: All you need to know
- FCT Training methods
- Evaluation metrics for FCT
- Datasets used in training FCT
- Conclusion
- What is forward compatible training (FCT)?: All you need to know
The notion of forward-compatible training (FCT) is derived from software engineering principles. software is easy to update since it is designed with the idea that it will be updated. In this case, we use a similar approach for neural networks. We plan to be compatible with some unidentified successor model when we train the outdated model because we anticipate there will be an update in the future. The architecture, goal, and training set for this model are all flexible. We suggest the idea of side-information or supplemental knowledge acquired at the same time as the previous model which makes it easier to update embeddings in the future. According to intuition, side-information captures data characteristics that may or may not be significant for the goal of training the current model but could be for the model's development in the future.
In the below image, we can see the demonstration of the concept. filtering red objects from blue objects is an old task. As color is the least discriminative property for this task, the previous model learns color as the feature for each item. Later, further information (triangles) is supplied, and the goal is changed to categorize objects according to both their shape and color. We are unable to discriminate between distinct forms that share the same hue, such as blue circles and blue squares, using simply the information that the old embeddings (color) carry. To help with future updates, we store side information with the old embeddings in FCT. The shape is the ideal side information to store in this situation.
- FCT Training methods
Forward-compatible training comes with a certain set of methods to train models. We will look at these methods which are used in FCT.
- Forward compatible setup
FCT has a few key properties and constraints:
1. In order to provide the new model with the maximum accuracy feasible, we do not alter its training.
2. Using a learned transformation, we reconcile the representation of the old and new models.
3. It might not be possible to directly convert from old embeddings to new ones. In order to prepare for a future upgrade, we store side information for each example in G when we train the previous model.
- Training FCT transformation
The transformation model comprises a mixing layer to reconstruct the new embedding and two projection layers representing the previous embedding and side-information branches.
- Training FCT side-information
For each example, the side-information encodes compressed features such that we can reconstruct new features from the previous ones.
- Decentralized design
In comparison to BCT, FCT entails two extra expenses: First, whenever a new image is added to the gallery set, side information must be computed and stored. Second, whenever the embedding model is updated, a transformation must be carried out. FCT can be implemented in a decentralized manner for numerous purposes. In such a setup, we have edge devices, each with its own gallery setup.
- Evaluation metrics for FCT
There are two evaluation metrics used in FCT training setups.
- Cumulative matching characteristics (CMC)
- Mean average precision(MaP):
MaP is a standard metric that summarizes precision and recall metrics by taking the area under the precision-recall curve.
- Datasets used in training FCT
To train the model with sufficient accuracy and precision, there is a number of image datasets are taken into training FCT models. below the datasets that were taken into training of FCT
- ImageNet-1k
- VGGFace2
- Places-365
- Conclusion
In this article, we briefly saw the introduction to forward compatibility training definitions and training setups. FCT is used to mitigate the drawbacks of independent representation learning and backward compatibility training. We also saw FCT training methods along with evaluation metrics and datasets that are used training process of FCT models.
