Hollander was born in Oxford, Oxfordshire, England, and attended The Dragon School and then Abingdon School. He read English at Selwyn College, Cambridge while actively participating in the many extracurricular theatre activities that Cambridge University offers. He was a member of the Footlights and the president of the Marlowe Society. Fellow student Sam Mendes directed him in several plays while they were at Cambridge, including a critically acclaimed production of Cyrano de Bergerac. After graduating from university and failing to secure a place at a drama school, Hollander was asked to play Macheath in the Donmar Warehouse's 1994 production of The Threepenny Opera.

Hollander's film and television appearances include Absolutely Fabulous, Martha, Meet Frank, Daniel and Laurence, Wives and Daughters, Gosford Park, The Lost Prince and Pride and Prejudice for which he received The Evening Standard Film Awards Comedy Award, and London Critics Circle Best Supporting Actor. He has worked repeatedly with Michael Gambon and Bill Nighy and is a good friend of James Purefoy. Although highly respected as a character actor and the recipient of several awards, many of his films will still poke fun at his height (5ft 5).

Hollander recently featured in parts II and III of the Pirates of the Caribbean trilogy. He is currently appearing on stage in Joe Penhall's play Landscape with Weapon at the The Royal National Theatre in London until June 2007.