It has happened in the past:
http://support.microsoft.com/default.aspx?scid=kb;en-us;833987
Microsoft’s GDI+ had a bug that allowed a form of buffer overflow into the heap. The heap is dynamically allocated by programs at run rime and typically contains executable code. A JPEG exploiting this vulnerability could overwrite internal program structures, altering the way the program runs.
