An academic conference is a typical conference set up where there are keynote speakers, and in front of the room facilitator.
A workshop has the sense of being a relatively small event with all participants involved in the discussion (although this is not always the case).
A symposium is similar to a conference, but there's ususally some sort of a casual involved, drinking, food, snacks, etc.
A congress is usually grand and more academic than workshop and symposium. It is probably larger than a conference.