West, A.G., Lee, I., Kannan, S., & Sokolsky, O. (2010). An Evaluation
Framework for Reputation Management Systems. Book chapter in 
Trust Modeling and Management in Digital Environments: From Social 
Concept to System Development (Zheng Yan, ed.), pp. 282-308. 
Information Science Reference, Hershey, PA, USA.


Abstract: Reputation management (RM) is employed in distributed and peer-to-
peer networks to help users compute a measure of trust in other users based on
initial belief, observed behavior, and run-time feedback. These trust values
influence how, or with whom, a user will interact. Existing literature on RM
focuses primarily on algorithm development, not comparative analysis. To remedy
this, we propose an evaluation framework based on the trace-simulator paradigm.
Trace file generation emulates a variety of network configurations, and
particular attention is given to modeling malicious user behavior. Simulation 
is trace-based and incremental trust calculation techniques are developed to
allow experimentation with networks of substantial size. The described
framework is available as open source so that researchers can evaluate the
effectiveness of other reputation management techniques and/or extend
functionality.

This chapter reports on our framework's design decisions. Our goal being to
build a general-purpose simulator, we have the opportunity to characterize the
breadth of existing RM systems. Further, we demonstrate our tool using two
reputation algorithms (EigenTrust and a modified TNA-SL) under varied network
conditions. Our analysis permits us to make claims about the algorithms'
comparative merits. We conclude that such systems, assuming their distribution
is secure, are highly effective at managing trust, even against adversarial
collectives.