In a default Ubuntu, and probably any other modern Gnome based Linux desktop, the Gnome keyring takes the role of the ssh-agent. If this is not desirable you can tell the keyring not to do that by setting the gconf variable /apps/gnome-keyring/daemon-components/ssh to false.
$ gconftool -s –type bool /apps/gnome-keyring/daemon-components/ssh false
At the next login you should see your environment variable SSH_AUTH_SOCK pointing towards a more proper socket. Note that the real ssh-agent is still started, assuming Ubuntu, thanks to /etc/X11/Xsession.d/90×11-common_ssh-agent.