Interested in psiTurk? Try out this quick start guide to running a simple experiment online! It steps you from installing to paying participants and should work for most people using updated versions of Linux or Mac OS X.
This guide uses a simple example experiment provided in the defaul psiTurk installation, but can be used to run any psiTurk experiment.
- Step 1: Install psiTurk
- Step 2: Create a default project structure
- Step 3: Enter credentials
- Step 4: Launch psiTurk in the new project directory
- Step 5: Start the server
- Step 6: Debug/view your experiment
- Step 7: Create a sandboxed HIT/Ad
- Step 8: Check your data
- Step 9: Monitor progress
- Step 10: Approve workers
- Step 11: Switch to “live” mode
psiTurk can be installed easily on any system that has the python package
$ pip install psiturk
psiTurk includes a simple example project which you can use to get started.
$ psiturk-setup-example Creating new folder `psiturk-example` in the current working directory ... Creating default configuration file (config.txt)
In order to get access to the Amazon Mechanical Turk features of psiTurk, you
need obtain and enter credentials for accessing Amazon Web Services. These
can be added to
~/.psiturkconfig. You can leave the
the default value.
$ cat ~/.psiturkconfig [AWS Access] AWS_ACCESS_KEY_ID = YourAccessKeyId AWS_SECRET_ACCESS_KEY = YourSecretAccessKey aws_region = us-east-1
psiTurk should be run in the top level folder of your project. You should be greeted with a welcome screen and command prompt.
There is also an extensive help system in the command line. Type
help to see a
list of available commands. Type
help <cmd> to get more information about a
particular command (e.g.,
$ cd psiturk-example $ psiturk welcome... psiTurk version 2.1.1 Type "help" for more information. [psiTurk server:off mode:sdbx #HITs:0]$
The psiTurk server is the web server which responds to external requests. To
start or stop the server type
server off, or
$ [psiTurk server:off mode:sdbx #HITs:0]$ server on Experiment server launching... Now serving on http://localhost:22362 [psiTurk server:on mode:sdbx #HITs:0]$
To debug or test the experiment, simply type debug. This will launch the default web browser on your system and point it at your experiment in a method which is helpful for testing.
Hint: If you are running on a remote server and want to disable launching the
debug -p (print only) which will print the debugging URL but
not launch the browser.
Altering the experiment code is beyond the scope of the quick start, but see this guide for details on how to modify and extend the stroop example.
$ [psiTurk server:on mode:sdbx #HITs:0]$ debug Launching browser pointed at your randomized debug link, feel free to request another. http://localhost:22362/ad?assignmentId=debugX12JJ8&hitId=debugA7NP2T&workerId=debugS9K039 [psiTurk server:on mode:sdbx #HITs:0]$
In order to make the experiment available to workers on Amazon Mechanical Turk you need to:
- Run your psiturk server on a machine that is publicly-accessible.
- Post a HIT on AMT, which will point MTurkers to your psiturk server address.
Use the ad_url settings to point to the location of your publicy-accessible experiment.
See the Running psiTurk on Heroku guide for an example of running your experiment on the webserver of a platform-as-a-service cloud provider.
The example below uses the Amazon Mechanical Turk “sandbox,” which is a place for testing your task without actually offering it live to real paid workers.
Run the following to post a HIT, and answer all prompts:
$ [psiTurk server:on mode:sdbx #HITs:0]$ hit create
Your HIT should now be visible on http://workersandbox.mturk.com if you search for your requester account name or the HIT title word “Stroop” (set in config.txt).
Important! Test to make sure that your Ad URL can be accessed from a place external to the network from which you created the HIT. If it cannot be accessed, then MTurkers won’t be able to access your HIT!
By default psiTurk saves your data to a SQLite database participants.db in your base project folder. You can check that everything is being recorded properly by opening that file in a SQLite tool like Base.
One simple way to monitor how many people have actually accepted your HIT is with
hit list --active or
hit list --reviewable commands.
This shows the HITid for each task, how many have completed, and how many are pending.
psiTurk provides many tools for approving workers, assigning bonuses, etc.
help hit and
One simple approach is to approve all the workers associated with a particular
HIT (once all the assignments are complete). To do this, use the
worker approve --hit <HITID> command.
$ [psiTurk server:on mode:sdbx #HITs:0]$ worker approve --hit 28K4SME3ZZ2MZI004SETTTXTTAG44LT Approving....
In order to create public hits on the “live” AMT site, you need to switch modes in the command shell using the mode command. To get back to the sandbox, just type mode again.
To avoid mistakes, psiTurk defaults to sandbox mode (this behavior can be changed in config.txt)
From here, everything is exactly the same as described for sandbox hits above.
$ [psiTurk server:on mode:sdbx #HITs:1]$ mode Entered live mode [psiTurk server:on mode:live #HITs:0]$