Bus Scene with Ed Norton and Brad Pitt

In the scene where both Ed Norton and Brad Pitt get on the bus, Norton says something like "I've been sizing everything up lately" then later in the scene a bum shoulder checks Norton, why doesn't he kick the bums ass?