Though not an elegant solution, it can be removed with jQuery currently.
grid.on('refresh',function(){
this.pager().widget().find(".ui-icon-refresh").hide();
})
I've added it as feature enhancements to make it easy to modify the pager via templates for next version.