...
Add more images/ROIs containing annotated and labeled areas of the labels that were poorly recognized to your training set.
If the annotated and labeled areas are correct and complete, include new annotated and labeled images/ROIs containing annotated and labeled areas in the training set.
Check if all areas in your validation images/ROIs were annotated and labeled correctly. If not, annotate and label all missing areas and correct any inaccuracies in your validation set.
...