Users complain server response is slow, but when I use the server everything is fine.

A. It could be the server Screen Saver! The Open GL screen savers (especially the pipes) can use every CPU cycle off the Server. In general you should always use the blank screen saver on a server.