If you want a sort of replacement for a blazer (left), worn to the office with a dress shirt and wool trousers, you need your sweater to remain trim and fitted. Something like lambswool is the way to go, balancing shape retention, durability and comfort.
If you want a light, airy knit to primarily wear poolside, thrown on like a towel with sleeves and tossed in the laundry, a summery linen blend is perfect.
If you're looking to upgrade from a sweatshirt or hoodie for weekend errands, something to pull on over a t-shirt with denim or chinos, cotton is perfectly fine. It might stretch out a little and that's OK.
And if the goal is comfort above all else, with softness and a relaxed fit that's basically one step up from a Snuggie™, where the primary accessories will be a crossword puzzle and a mug of coffee, that means cashmere (right). It might pill up but hey, you're probably asleep by now; you won't care.
Shown left to right: Polo, J.Crew, Lands' End Canvas, L.L.Bean Signature.