Currently, I am examining the abilities of transformer models to generalize both in-domain (interpolation) and out-of-domain (extrapolation). Most previous work has only been interested in extrapolation over the input length. My experiments will examine extrapolation by defining the input space in a way that we can measure distances between the inputs. In doing so, I will be able to train the model in a certain input domain and then query it with inputs increasingly far away from it. Using this distance measure, I can also find out how model performance depends on how densely the input space was sampled during training.

Further links

• GitHub