Hi Dawie,
You feeling is correct. Just one reason is that they will be at different states of charge as they have different capacities and a single charge controller won't control the charging process correctly across both batteries.
I would remove the smaller capacity battery and just leave the larger. If you need the additional capacity replace the smaller battery with one identical to the larger one.
Cheers,
Martin.