The recommendations for X amount of fruits and vegetables a day are a way for the general public to focus on healthier non-processed sources of carbohydrates, more specifically, the focus is on making sure people get their required levels of micronutrients and fibre which processed carbohydrates dramatically lack.
Does this mean I shouldn't be worried? I rarely eat processed food. My main sources of carbs are oats, fruits and vegetables.