![[Rounds decrease steadily]](rounds.gif) |
As the community of actors gets better at coordinating their behavior for regularly occurring problems of coordination, one would expect to see some general improvements in the performance of the community.
One statistic that characterizes the overall performance of the group of actors is rounds of activity. A round occurs each time all of the actors have completed a UNIT of activity. A unit is constituted by either a conversation between two actors or an attempt to execute one of the primitive actions available to actors in MW, for example UNLOAD-TOGETHER or PUSH-HANDTR. Occasionally, an actor will not attempt any action (technically, they execute a NO-OP) because their planner has failed to generate a plan [this occurs between 1.0 and 2.7 times per problem].
Figure 8 shows improvements in group performance with and without actors learning from prior experience of coordinated behavior. The average number of rounds required to solve these problems before learning conventions was 38.6, with a standard deviation of 0.71. Even without procedural memory, there is improvement in group performance; these improvements result from increasingly accurate probability estimates. The addition of procedural memory leads to statistically significant improvement; the average number of rounds drops fairly steadily, ending at 21.2, a 49% improvement over the baseline system. Standard deviations for points after the second problem-solving episode ranged from 0.52 to 0.66, so the small improvements along the tails of the learning curves are not significant.
Overall system performance of an AI system is often measured by CPU usage. However, CPU usage is an inappropriate measure of community performance in our domain because we are primarily interested in improving the community's runtime behavior, not the speed at which they plan (or retrieve plans from memory).1