psiTurk is free, open source software provided to scientists to aid in research. Because it helps you run paid experiments online using Amazon Mechanical Turk errors in the software, or in your use of the software, can lead to loss of money. This is the very nature of online research (errors may mean someone will actually do your task and you need to pay them as a result).
Our belief is that these types of errors can be best limited by having open, peer-reviewable software and sharing bug reports between labs and research groups. In other words, even if you wrote this software yourself it is possible that some bug could cost you money when getting started.
We take no responsibility for your use of the software. We make no claims that it is bug-free and any errors are not our responsibility. This is a community-run, community-supported system and not a company selling a product. We use the software in our lab and, when used correctly, has never caused us to lose money on Mechanical Turk due to mistakes. However, it is always possible to mis-use the software in a costly way.
In addition, while we strive to keep the psiturk.org Secure Ad Server running, crashes in that system could, in the short-term, affect your ability to collect data. Again, using the system you must understand what the risks are. The good news is that because the system is open source if there is a problem everyone can read the code themselves and make suggestions on how to fix things.
Some suggestion to avoid costly mistakes from happening are
1. Test your code a lot in the sandbox to make sure every stage is working and you understand what psiTurk is doing.
- Run small batches at a time to verify everything is working
3. Keep your Amazon payments account balance reasonably low at any point in time. It is impossible to spend more money than is in your account at any point in time.
4. Exit the psiTurk server when you are not using it to collect data (i.e., do not leave psiTurk server running indefinitely). This ensures that no one will be able to actually perform your task and then claim they are owed payment. This also limits the ability of bots and other scammers to reverse engineer your task.
5. When testing “live”, explain in the text of your Ad that this is a test and you are looking for feedback. Workers get fustrated when you put bad or broken experiments online, but are often very helpful if you explain that you are hoping to get feedback on an unfinished project.