Hello, I am a bit puzzled why you would be having unpublished products showing up in the feed. Which version are you using? Which version of Joomla and Virtuemart?
I will run a test for this in case there is a problem, but I am fairly sure that there is not. The feed should only pick up products that are published in Virtuemart, the code definitely checks whether the product is published. However it will still display products that are in a category that is unpublished. If you leave a product itself published and just unpublish the category then the product will still be picked up.
Could it be a caching problem? Virtuemart caches a lot of product information, also Joomla and web browser caching might cause the feed not to be updated when you unpublish the product. You should definitely clear all caches and check if the problem goes away.