Custom loss for classification of geolocation in Keras

I have a model that predicts geolocation coordinates based on some data. The way I have it set up at the moment is that I clustered my points (2D coordinates) into 100 classes that my model predicts with categorical crossentropy as the loss. I realized that this penalizes predictions of (phisically) neighbouring classes just as much as those that are far apart. Ideally my loss would be the euclidian distance between the prediction and the class centroid but then my model would have to read points from a list depending on a layer's output and I'm not sure how to do that (and it seems inelegant).

What would be the proper way to approach this problem?