Update: I noticed that the screen resolution of my OS X guest VM is changed properly as soon as the window dimension reaches a level of 1024x768 or higher. As soon as the window dimension would lead to a lower guest resolution, it is not updated at all, and the screen turns blurry. Can anybody confirm this?
This seems to affect OS X guests only. Within Windows or Linux VMs the resolutions are also scaled to lower values than 1024x768. I was wondering how practically this is, as Low DPI guests are likely requiring lower resolutions on Retina displays to fit on the High DPI screen in window mode.