The two main metrics we are focusing on with respect to distributed organizations are robustness and responsiveness.
Examples: Bio-inspired organizations and swarms are examples of robust organizations.
A Robust system should overcome these without much performance decrease.
Sensitivity to these abnormalities can be measured as the change in efficiency with respect to the performance criteria.
Additional measurements that should be low for robust systems.
Examples: Hierarchical organizations are usually very responsive.
Things a hierarchical organization should have.
Ways to increase efficiency in hierarchical organizations.
From Sujit's Experiments.
Robustness seems to need to be measured with respect to a type of unexpected phenomena. A few examples of unexpected phenomena are listed above. Since performance can be measure using a number of metrics, each performance metric can be tested for its robustness to a type of unexpected phenomena. The graph of the robustness of a performance metric w.r.t. the degree of an unexpected phenomenon will result in an unreliability inpact curve analogous to a neglect impact curve. From the board, we called this sensitivity. But perhaps robustness is the sum total of such sensitivity curves.
The word responsiveness seems like it should measure how well the system responds to some change, and not just system performance. Responsiveness could be measured as the time it takes to reach a type of organizational, informational, or environmental equilibrium after a task, role, goal, or organization change.
Perhaps performance should itself be a separate metric class.
From “Developing Performance Metrics for the Supervisory Control of Multiple Robots” by Jacob W. Crandall and M. L. Cummings. (HRI2007)
From “Identifying Generalizable Metic Classes to Evaluate Human-Robot Teams” by P. Pina, M. L. Cummings, J. W. Crandall, and M. Della Penna. (HRI2008)
The metric classes in the earlier paper listed seem to be represented in the latter.
It seems correct to split the behavior of the robot and human into different metrics. Collaborative Metrics could also be split into human-human, human-robot, and robot-robot situations. This could help identify (or predict) the location of a problem, whether it is a human problem (“problem exists between keyboard and chair”), an autonomy problem, or an organizational problem.
“Entropy of activity, entropy of information, .. ” “The idea is that entropy should have a time-varying quality that satisfies some pattern.” “I'm not sure what that pattern should be.”
A few possible types of entropy for metrics:
What types and measurements of entropy would be useful in these situations?
Informational entropy over time could be a good way of measuring information flow and how information is distributed among agents.
Behavioral entropy could be a good way of measuring a human's state (from driving and measuring how sleepy people are).
What would organizational entropy be? Would it require organizations to self-organize?