The Puritans were sticklers for taking the Bible, and nothing else, as their guide for how to live a good, Christian life. And since the Bible never indicates exactly when the anniversary of the Nativity should be observed, they reasoned that God must not have intended for it to be observed at all. Otherwise a date would have been provided. So they banned its celebration. Between 1659 and 1681 it was actually a criminal offense, punishable by a fine of five shillings, to celebrate Christmas in Massachusetts.
*********
Christmas only became a legal holiday in England and America late in the nineteenth century. Before then people were expected to go to work on Christmas Day.

19th-century factory owners didn't like having to give their employees a day off work, so they long lobbied against having to treat Christmas as anything but a regular working day. But apparently some kindly factory owners would generously let their workers start work at 5 am on Christmas Day, thus allowing them to get off work early and go to church.