Federal election times are set by 2 U.S. Code § 7 as 1 day after the 1st Monday in November (of even numbered years). The law is from 1875 and from what I can tell is indeed nominally motivated by the voters’ need to first observe rest day on Sunday and then travel to their polling place. Keeping it and not having a federal holiday coinciding with it is largely aimed at keeping voter turnout low.
“Have they no refuge or resource?” cried Scrooge.
“Are there no prisons?” said the Spirit, turning on him for the last time with his own words. “Are there no workhouses?”