Testlet-based format is widely used in standardized tests and classroom assessments.  This format, however, poses a challenge to reliability and item parameter estimation.  We examine the effects of testlet on item difficulty estimates with science assessments for three units at Grade 5.  The research questions are: Is there any testlet effect on item difficulty estimates across three units? If yes, what characteristics of the shared stimulus may be linked to the testlet effect? We employ three methods with Winsteps and IRTPRO: (1) assuming no testlet effect - the Rasch model with individual items; (2) assuming testlet effect - the polytomous IRT using accumulated testlet scores; and (3) assuming testlet effect - the testlet response theory model. We then study the characteristics of the flagged items with the testlet effect.

Our preliminary analysis based on the first two models was conducted with items from one science unit: My Skokomish River Challenge.  Using the Rasch model, item difficulties ranged from easy to moderate (i.e., -1.86 to .54).  Using the polytomous IRT, item difficulties were in the easy range (i.e., -1.45 to -.49).  Testlets 2 and 4 yielded poor model fit statistics which might be associated with the testlet effect.  Considering item characteristics, items with high correlations from these testlets were in the constructed-response format and measured exactly the same learning target (e.g., understanding and reasoning of the impact of the topographic map of floods).

The preliminary findings will be verified by including the result from the testlet response theory mode and additional analysis with two other units.  Items in the same testlet that ask for vocabulary or understanding of the same scientific phenomena tended to have high correlation, thus resulting in local item dependence.  Final paper will provide recommendations regarding model selection and item development for testlet-based items.