With the M size, the collar felt tight and uncomfortable when I buttoned the top button.
On the other hand, the XL size is a bit too roomy and looks a little loose.
Ultimately, it comes down to personal preference, but I chose the L size because it has the most flattering overall silhouette.