State transition diagrams have been used right from the beginning in object-oriented modeling. State transition diagrams give an explicit, even a formal definition of behavior. The basic idea is to define a machine that has a number of states (hence the term finite state machine). The machine receives events from the outside world, and each event can cause the machine to transition from one state to another. Thus you can get a good sense of what events should occur, and what effect they can have on the object.
A big disadvantage is that you have to define all the possible states of a system. While this is all right for small systems, it soon breaks down in larger systems as there is an exponential growth in the number of states. This state explosion problem leads to state transition diagrams becoming far too complex for much practical use.
To combat this state explosion problem, object-oriented methods define separate state transition diagrams for each class. This pretty much eliminates the explosion problem since each class is simple enough to have a comprehensible state transition diagram. (It does, however, raise a problem in that it is difficult to visualize the behavior of the whole system from a number of diagrams of individual classes - which leads people to interaction and activity modeling).
A particularly valuable feature of the approach is its ability to generalize states, which allows you to factor out common transitions.
Processes that are instantaneous (i.e. cannot be interrupted) can be bound to the transitions or to the entry or exit of a state; these are called actions. Processes that are long (and can be interrupted) are bound to states; these are called activities. Transitions can also have a condition attached to them, which means that the transition only occurs if the condition is true. There is also a capability for concurrent state diagrams, allowing objects to have more than one diagram to describe their behavior.
State models are ideal for describing the behavior of a single object. They are also formal, so tools can be built which can execute them.
Their biggest limitation is that they are not good at describing behavior that involved several objects. For these cases use an interaction diagram or an activity diagram.
People often do not find drawing state diagrams for several objects to be a natural way of describing a process. In these cases you can try either drawing a single state diagram for the process, or use an activity diagram. This defines the basic behavior, which you then need to split across a number of objects.