Lets face it, the road surface are always dirty. Plenty of road gunk and muck would bring up to the rim’s brake track during the ride. These kind of dirt would reduces braking performance and speeds up the wear on the rims and the pads.
During the service of the bicycle, you can apply bike cleaning formula to the rim surface. If it get really worse, you can try the brake track cleaning rubber to do the job. Within the matter of minutes, the brake surface could be as good as new.