You'll have more control over your schedule with a rental car. Cheapest route is by train; Airtrain from terminal to Jamaica Station, either E subway or LIRR to Penn Station and NJ Transit to New Brunswick (approx. $25 total). Allow about three hours for train travel time, train transfers, etc. and be at JFK two hours before your return departure. See www.njtransit.com for train schedules and fares. If using a rental car be sure to arrive at JFK rental car return by 6am so you can be at your terminal by 6:30am. Allow at least 1.5 hours driving time. Rental cars are located at the Federal Circle stop on Airtrain, only takes about 10 minutes from there to terminal 5.
You'll have more control