The conference abstract review process is a critical component of planning an event. A well-designed and executed peer review makes it easy to select only the best papers and presentations for your event.
There are two areas of the review process we see meeting planners and program committees struggle with most often:
- Getting reviewers to participate and complete their assigned submissions
- Having enough quality input to make confident accept/reject decisions
To help you minimize these challenges, we’ve compiled a list of tips and tricks provided by our panel of abstract management professionals who work with hundreds of conferences each year.
Setting up your abstract review form
Start by defining your review criteria
It’s important to make sure you have fully defined the review process and guidelines before you begin setting up your review form. This will save you time and help ensure your process supports the standards of your conference. Here are some of the questions you’ll want to consider:
- What are your criteria for paper or speaker acceptance?
What does a “good” submission look like? This will help you identify the questions you need to ask and the scoring rubric to use.
- Will it be helpful to understand the context behind a reviewer’s score?
If so, then be sure to add a comment box on your form. If you do, provide specific direction on what type of input you’re looking for.
- If the reviewer can see the submitter’s identity, will this help or hinder the review process?
This will inform whether you need to set up a blind or double-blind review process. There is no right answer to this question. It depends upon your industry and conference.
- How many times do you want each submission to be reviewed, and by whom?
How much input do you need to make a confident accept or reject decision? This will help you determine not only how many reviewers to recruit, but how many rounds of review to set up in your abstract management system.
- How are you assigning reviews?
Do you want to assign a submission to a specific reviewer manually, or auto-assign submissions by matching your reviewers to a topic area? The answer to this question will help shape your recruitment process, making sure you have the right mix of reviewers to cover all submission categories.
Here’s an extra tip:
On your submission form, did you put specific guidelines in place that you wanted your submitters to follow like word counts, file types, etc.? If general compliance is important to your selection process, then include it on your review form. Let your reviewers tell you whether a submitter provided all requested information, in the format requested.
Keep the review form simple and concise.
You want to make it as easy as possible for your reviewers to understand the expectations and indicate whether a submission meets the stated criteria.
Scoring that is overly complex – for instance rating scales that are also weighted – makes it more difficult for your reviewers to provide clear and complete input. Reporting also becomes more complex, which can complicate the selection process.
And include context wherever possible. For example, if you are providing a rating scale, add descriptive labels to the numbers so the reviewer understands the difference between each value.
Test the abstract review form
After you’ve finished setting up your review form, test it before inviting reviewers into your abstract management system. Your reviewers have limited time, and you want to make sure the form is easy to understand and navigate. This will also give you greater confidence in the quality of reviews you receive.
For tips on how to conduct user testing, check out this article, Everything You Need to Know About Testing Your Abstract Management System.
Recruiting and managing your reviewers
Recruit more reviewers than you think you’ll need
It’s likely that some of the reviewers you recruit will have conflicts, be unresponsive, or provide only a fraction of the feedback required. To avoid being shorthanded, recruit a few more reviewers than you think you’ll need at the beginning of your call for papers process. Having the ability to fill any gaps in your reviewing team will ensure your content is properly vetted without any major delays.
Check their review history
For repeat reviewers, check their review history before inviting them back. Did they complete their reviews in the past? Were they quality reviews? If the answer is no, you may want to reconsider inviting them back again this year.
Provide concrete expectations up-front
As part of your recruitment process, quantify up-front the estimated number of reviews they’ll have to complete, how many questions they’ll have to answer, approximately how much time it should take per review, and whether they will need to have the ability to download a paper or other file. Be very specific on the expectations so prospective reviewers truly know whether they can fulfill their responsibility.
Only assign complete submissions
If you abstract management software allows submitters to save and come back to an in-progress submission, this tip is important. If you pull a report on all submissions in the system, these in-progress submissions may show up and be assigned to a reviewer who won’t be able to complete the review.
Make sure you’re running a report that includes submission status, so you only assign completed submissions. It’s a simple step, but one that is often overlooked.
Communicate to reviewers early and often.
If you tend to use the same pool of reviewers each year, make sure to confirm each year whether they can participate before you plan your review assignments. This will help cut down on abandoned review assignments. And, even more importantly, confirm their email address every time.
Some organizations have a longer submission timeframe. Which means you may have recruited your submitters months before you need them. During this time, things may have changed. It’s always important to check in with your recruited reviewers before you start assigning submissions to double-check their availability.
And make sure to check in on the progress of each reviewer’s assignments on a regular basis. A general rule of thumb is to do this weekly while the review round is open. If you are using an abstract management software, this task should only take about 15 minutes a week. You should be able to quickly run a report to see how many submissions each reviewer has and the status of those submissions. You should also be able to schedule and send emails through your system to reviewers who have not completed their assignments.
Your reviewers have the best intentions, but things do come up. And they may not have the time they thought they would. Checking in frequently allows you to re-assign submissions early enough so that you don’t compromise either the quality of the review or your timeline.
Looking for more ideas, best practices and tips on managing a call for abstracts, papers, or speakers? Visit our collection of abstract management articles and resources.
Share The Love!