Image classification model only predicting one class

I'm trying to build a deep learning model to predict image classes from the Kaggle competition. I'm using the Xception model as the top layers and then putting the last layer into a dense layer with the softmax output of 4. No matter what I try with the model, it only predicts one class, the first-class 'healthy', even on the training set which gets a high accuracy. The data set is imbalanced in the multiple_diseases class, but even when I use RandomOverSampler, I get the same problem. There has to be an error in the way I am uploading my images or feeding them into the model. predict function. Anyway, here is the code:

healthy: 516 multiple_diseases: 91 rust: 622 scab: 592

train_df = pd.read_csv('train.csv')
test_df = pd.read_csv('test.csv')
targets = train_df[['healthy', 'multiple_diseases', 'rust', 'scab']]

print(train_df.describe())
print(test_df.describe())

           healthy  multiple_diseases         rust         scab
count  1821.000000        1821.000000  1821.000000  1821.000000
mean      0.283361           0.049973     0.341571     0.325096
std       0.450754           0.217948     0.474367     0.468539
min       0.000000           0.000000     0.000000     0.000000
25%       0.000000           0.000000     0.000000     0.000000
50%       0.000000           0.000000     0.000000     0.000000
75%       1.000000           0.000000     1.000000     1.000000
max       1.000000           1.000000     1.000000     1.000000

from keras.preprocessing.image import load_img
from keras.preprocessing.image import img_to_array
from tqdm.notebook import tqdm

path = '/content/images/'
num_img = train_df.shape[0]
size = 224
train_images = np.ndarray(shape=(train_len, size, size, 3))
    for i in tqdm(range(num_img)):
      img = load_img(path + f'Train_{i}.jpg', target_size=(size, size))
      train_images[i] = np.float32(img_to_array(img))

test_images = np.ndarray(shape=(test_len, size, size, 3))
    for i in tqdm(range(num_img)):
      img = load_img(path + f'Test_{i}.jpg', target_size=(size, size))
      test_images[i] = np.float32(img_to_array(img))

from keras_preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(rescale = 1./255,
      rotation_range=40,
      horizontal_flip=True,
      vertical_flip=True,
      fill_mode='nearest')

validation_datagen = ImageDataGenerator(rescale = 1./255)

train_generator = train_datagen.flow(
    x = x_train, 
    y = y_train,
    batch_size = 32
    )

validation_generator = validation_datagen.flow(
    x = x_test,
    y = y_test,
    batch_size = 32
    )

def create_model():
  pretrained_model = tf.keras.applications.Xception(input_shape=[*[224, 224], 3], include_top=False)
  pretrained_model.trainable = True
  model = tf.keras.Sequential([
      pretrained_model,
      tf.keras.layers.GlobalAveragePooling2D(),
      tf.keras.layers.Dense(4, activation='softmax')
      ])

  model.compile(
        loss = 'categorical_crossentropy', 
        optimizer = 'adam', 
    metrics = ['accuracy'])
  return model

model = create_model()

model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param    
=================================================================
xception (Model)             (None, 7, 7, 2048)        20861480  
_________________________________________________________________
global_average_pooling2d (Gl (None, 2048)              0         
_________________________________________________________________
dense (Dense)                (None, 4)                 8196      
=================================================================
Total params: 20,869,676
Trainable params: 20,815,148
Non-trainable params: 54,528
_________________________________________________________________

epochs = 12
batch_size = 32
steps_per_epoch = 55

start_lr = 0.00001
min_lr = 0.00001
max_lr = 0.00005
rampup_epochs = 5
sustain_epochs = 0
exp_decay = .8

def lrfn(epoch):
  if epoch  rampup_epochs:
    return (max_lr - start_lr)/rampup_epochs * epoch + start_lr
  elif epoch  rampup_epochs + sustain_epochs:
    return max_lr
  else:
    return (max_lr - min_lr) * exp_decay**(epoch-rampup_epochs-sustain_epochs) + min_lr

lr_callback = tf.keras.callbacks.LearningRateScheduler(lambda epoch: lrfn(epoch), verbose=True)

history = model.fit(
    train_generator,  
    validation_data = validation_generator,
    epochs = epochs,
    steps_per_epoch = steps_per_epoch,
    validation_steps = steps_per_epoch,
    callbacks = [lr_callback],
    verbose=1)

Epoch 00001: LearningRateScheduler reducing learning rate to 1e-05.
Epoch 1/12
55/55 [==============================] - 25s 454ms/step - loss: 1.3612 - accuracy: 0.3166 - val_loss: 1.2711 - val_accuracy: 0.4729 - lr: 1.0000e-05

Epoch 00002: LearningRateScheduler reducing learning rate to 1.8000000000000004e-05.
Epoch 2/12
55/55 [==============================] - 24s 435ms/step - loss: 1.1027 - accuracy: 0.6805 - val_loss: 0.8197 - val_accuracy: 0.7671 - lr: 1.8000e-05

Epoch 00003: LearningRateScheduler reducing learning rate to 2.6000000000000002e-05.
Epoch 3/12
55/55 [==============================] - 24s 435ms/step - loss: 0.6554 - accuracy: 0.8310 - val_loss: 0.4950 - val_accuracy: 0.8487 - lr: 2.6000e-05

Epoch 00004: LearningRateScheduler reducing learning rate to 3.4000000000000007e-05.
Epoch 4/12
55/55 [==============================] - 24s 435ms/step - loss: 0.3981 - accuracy: 0.8714 - val_loss: 0.4149 - val_accuracy: 0.8696 - lr: 3.4000e-05

Epoch 00005: LearningRateScheduler reducing learning rate to 4.2000000000000004e-05.
Epoch 5/12
55/55 [==============================] - 24s 434ms/step - loss: 0.3024 - accuracy: 0.8991 - val_loss: 0.3442 - val_accuracy: 0.8785 - lr: 4.2000e-05

Epoch 00006: LearningRateScheduler reducing learning rate to 5e-05.
Epoch 6/12
55/55 [==============================] - 24s 436ms/step - loss: 0.2155 - accuracy: 0.9291 - val_loss: 0.3372 - val_accuracy: 0.8910 - lr: 5.0000e-05

Epoch 00007: LearningRateScheduler reducing learning rate to 4.2000000000000004e-05.
Epoch 7/12
55/55 [==============================] - 24s 436ms/step - loss: 0.1778 - accuracy: 0.9435 - val_loss: 0.2799 - val_accuracy: 0.8958 - lr: 4.2000e-05

Epoch 00008: LearningRateScheduler reducing learning rate to 3.5600000000000005e-05.
Epoch 8/12
55/55 [==============================] - 24s 436ms/step - loss: 0.1446 - accuracy: 0.9516 - val_loss: 0.2872 - val_accuracy: 0.9041 - lr: 3.5600e-05

Epoch 00009: LearningRateScheduler reducing learning rate to 3.0480000000000006e-05.
Epoch 9/12
55/55 [==============================] - 24s 436ms/step - loss: 0.1359 - accuracy: 0.9539 - val_loss: 0.2637 - val_accuracy: 0.9017 - lr: 3.0480e-05

Epoch 00010: LearningRateScheduler reducing learning rate to 2.6384000000000004e-05.
Epoch 10/12
55/55 [==============================] - 24s 435ms/step - loss: 0.1164 - accuracy: 0.9596 - val_loss: 0.2602 - val_accuracy: 0.9130 - lr: 2.6384e-05

Epoch 00011: LearningRateScheduler reducing learning rate to 2.3107200000000005e-05.
Epoch 11/12
55/55 [==============================] - 24s 435ms/step - loss: 0.1209 - accuracy: 0.9723 - val_loss: 0.2699 - val_accuracy: 0.9124 - lr: 2.3107e-05

Epoch 00012: LearningRateScheduler reducing learning rate to 2.0485760000000004e-05.
Epoch 12/12
55/55 [==============================] - 24s 435ms/step - loss: 0.0896 - accuracy: 0.9758 - val_loss: 0.2449 - val_accuracy: 0.9113 - lr: 2.0486e-05

probabilities = model.predict(test_images)
print(probabilities)

[[1.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00]
 [1.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00]
 [1.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00]
 ...
 [1.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00]
 [1.000000e+00 0.000000e+00 0.000000e+00 0.000000e+00]
 [1.000000e+00 0.000000e+00 0.000000e+00 5.321019e-38]]

Same thing for training images:

idx = 500
tester = np.reshape(train_images[idx], (1, size, size, 3))
print(tester.shape)
print(train_df.iloc[idx,:])

probabilities = model.predict(tester)
print(probabilities)

[[1. 0. 0. 0.]]

And it does that for all the training images. What is going on here?

Edit: Including a description of the data set.