Technically thing that communicates with a monitor and thing that crunches vectors, textures and polygons are completely different and separate devices.
We just stick them on one circuitboard because of historical accident/convinience.
What would be cool is to separate them back, throw away "graphics" part of graphics accelerator and make it into general purpose parallel processing accelerator, then reimplement OpenGL in OpenCL. Or implement Vulkan/Mantle in OpenCL. Or let every little program ship with it's own graphics library.