Can I specify the order in which payees appear in the payee groups?

Yes, you can sort them by Payee Lookup Name, Pay From account, Payee Name, Bank Identifier, Payee Account Number, Payee Reference, Payment Type or Status by clicking column name and then on the arrow to change from ascending to descending order.