The arithmetic of the Hitchin component