Questions we Can Help you to Answer

Paper Instructions:

Conducting Your Tryout

Formative evaluation is the process of trying out instruction with learners with an eye toward improving it over time. When you try out your instructional materials with members of your target audience, you will be doing the first round of what should be several cycles of tryouts and revisions. We have time for one, so that's what we'll do.You will try out the instruction with a very small number of learners. You will not actually make the revisions, just outline what revisions are needed, based on the tryout results. 
Preparing for the Tryout

You should make sure you are all set before you begin the tryout itself. This can mean several things:

    Have all the instructional materials ready, with copies for each student and at least one for yourself. You should have a copy so that you can follow along and take notes.
    Have your data gathering materials ready. These include the tests and assessments that you will give the students, follow-up surveys if you have them, and interview questions. Note that this is when you decide what tests and assessments to use. If at all possible, you want to give the prerequisites test, the pre-test, and the post-test to all students, especially in the tryouts. Otherwise, how will you know what they did or did not learn? The best tryouts gather as much data as possible. Once you have the unit working the way you want, you can eliminate some of these things. There are some reasons for not giving all of this; check your book for some of them or discuss the issue online.
    Have an appropriate place for the tryout ready. For this first round, you do NOT have to duplicate the classroom setting. A quiet place, free of distractions, is best.
    Recruit the learners. Try to have them be members of the general target audience. Since we are doing "one-on-one" tryouts in this first round, you will need three (count 'em, 3) learners. One or two is not enough, while more than three just makes more work for yourself. An exception might be if you have group work requiring more. You can either choose the three learners at random or deliberately try to get one good student, one average one, and one low-achiever from your group. Do NOT use all great students or all of any other identifiable group.

Conducting the Tryout

Run each student individually through the tryout unless there is good reason not to (check with me). Collect lots of information along the way. Here's what you do:

    Bring each student into the tryout setting. Try to make sure that he or she is at ease.
    Give them any pre-instruction assessments that you might require. Remind them to do their best, but note that they aren't always expected to know everything there. If your assessments and instruction are long, then you can give the assessments at different times from the instruction itself. This can be especially useful with young children, who may tire and lose focus. 
    Start them on the instruction itself, and ask them to "think aloud" while doing it. In your copy of the materials, note the start time.
    As they go through the materials, write down any significant comments and questions that the students have while thinking aloud (you might have remind them occasionally to think aloud). At major section breaks, note the time again. If they look puzzled or unsure, ask them what they are thinking. If they hit a snag and cannot seem to move forward, then in this tryout you ARE allowed to help them. Make sure that you write down where and when the help occurred, as well as what it was.
    If students are doing practice exercises, etc. during the unit, you should either make sure that they write the answers in their materials or note them in your copy
    After the learners complete the instruction, note the ending time.
    Give them the post-instructional assessment. Again, this could be done at another time, if necessary. 
    After they have completed all this, you should try to gather some data on their reactions to the unit. This can be done either with a short survey, an interview, or both. Have some questions ready for the interview.
    Thank the learners for their participation.

---------------------- First two pages ----------------------

Writing up the Tryout

After you have completed the tryout, you need to analyze the data and write up the results. The general outline of the report is found  in the assignment for this section. Here are some other things to keep in mind:

    Data analysis for a three-person tryout cannot and should not include a lot of statistical analysis. It can, however, include tables and so on with summaries of all the basic findings. What are the scores on the tests? What about breaking that down into the different objectives? As people moved through the materials, did they get all the practice items right, or were there variations?
    There is also qualitative data. Where did people have the most trouble with the materials? What were their comments during the instruction? What questions did they ask? Did you have to help them over any hurdles? What did they say afterwards?
    Your goal for the report is NOT to dump everything on me. For example, I do not want the original tests from each students. You do want to have a complete summary of all the data which paints a picture of what worked and what didn't.
    At this point you should just be reporting, not drawing conclusions, being defensive, etc.
---------------------- Last two pages ----------------------
Revising the Materials

You are not required or expected to make any actual revisions to your project during this semester. There just isn't time. You are, however, expected to tell me what the needed revisions are. There can be a wide range. Here are some examples:

    Formatting changes to make sure that people can tell what it important, what tasks they should be doing, and so forth.
    Typos and wording changes needed to make things as clear as possible.
    Scope changes where you find that your project either included too much or didn't include enough. Did your students not have the prerequisites? Then you might discuss how to include instruction in those areas. Did they already know half of the material? Then you might have to revise the analysis and the instruction to reduce its size.
    Strategy changes in which you discuss why your chosen instructional strategies didn't work well and how you should redesign the project.
    And many more....