If decision is reversed at any point, you are required to repay all that has been paid to you.
This is what NYS says:
Unemployment Insurance Early Repayment Program - New York State Department of Labor
NYS will also probably participate in the Treasury Offset Program which will withhold any tax refunds and instead forward them to your state for repayment of your overpayment. It will also not pay benefits on any future claims until the overpayment from the previous claim is paid.
Last resort would be wage garnishments and liens - but usually the states don't go that far until they've tried everything else. At that point, you should call to set up payment arrangements.