关键词：众包劳动力；视觉设计批判；接口
摘 要：Crowdsourced labor has successfully tackled a variety of rote tasks like image tagging and transcription, but has difficulty accomplishing complex tasks. Visual design critique is a crucial part of the design process for improving upon existing work, but it is a complex task that requires some implicit knowledge of design concepts on the part of the critiquer. In this report, we propose a model for crowdsourcing visual design critique to enable online workers who lack background knowledge to provide useful feedback on designs. Our model organizes the “crit” around a set of design principles, and asks workers to select relevant statements about design principles and provide freeform comments within the scope of the selected statements. We
evaluate two variants of a critiquing system we built based on this model (CrowdCritter) through a study with 73 participants. We found that the integration of a set of design principles allowed novice workers to provide more critiques than freeform feedback alone. We were unable to identify significant differences between the two variants of CrowdCritter in the quality and level of agreement of critiques produced through each interface.
目 录：

ABSTRACT

INTRODUCTION

BACKGROUND: FROM TRADITIONAL CRITIQUES TO ONLINE CRITIQUES

RELATED WORK

FORMATIVE PILOT STUDIES

PILOT #1: NA?VE VISUAL FEEDBACK

PILOT #2: STRUCTURED FREEFORM VISUAL DESIGN FEEDBACK

PILOT #3: STRUCTURED CHECKLIST VISUAL DESIGN FEEDBACK

THE CROWDCRITTER SYSTEM

CONSTRUCTING A SET OF STRUCTURED DESIGN PRINCIPLES

INTERFACE WORKFLOW

CRITIQUE-FIRST INTERFACE

FREEFORM-CRITIQUE INTERFACE

IMPLEMENTATION

EVALUATIVE STUDY

STUDY DESIGN

RESULTS

QUALITY

QUANTITY

BREADTH

AGREEMENT

DISCUSSION

CONCLUSION AND FUTURE WORK

REFERENCES

APPENDIX A: DESIGN PRINCIPLES

APPENDIX B: CRITIQUE QUALITY GRADING RUBRIC

APPENDIX B: DESIGNS IN EVALUATIVE STUDY SMALL POOL