A major challenge for the past 25 years in educational assessment has been the need to individualize learning in classrooms - and in using the tools now at our disposal, to focus on learning and growth over time. This in Learning Oriented Assessment (LOA) often means using mini e-assessment during a course of language learning. E-assessments would allow learners receive individualized feedback based on their performance with recommendations directing them to development activities in the form of remedial, development or extension exercises depending on their test scores. As the learning programme progresses more classroom test data is collected and an electronic student profile emerges, and saved in the form of an electronic portfolio. Formal e-records of achievement can be maintained by individual learner or for the whole class. This in turn has the potential to inform teacher planning, feeding back into the teaching cycle.

The field of automated assessment of text focuses on automatically analyzing and assessing the quality of writing. A number of systems have been developed, usually employing natural language processing (NLP) and machine learning techniques, where language learners can submit samples of their writing and receive feedback on them. Some (e.g. Conference on College Composition and Communication, 2009) have criticized the use of these systems as being ultimately counter to good writing and instructional practices. Indeed, contemporary approaches to validity and validation require developers of assessment instruments to account not only for the construct they are measuring, but also for their use, consequences, and impact.

In this paper we discuss the development of a Write and Improve online system that can be used to provide instantaneous diagnostic feedback within a learning-oriented, self-access, and/or teacher-directed assessment context, and address issues of construct validity, as well as of assessment use and impact. The system, using NLP, maps learners’ output to proficiency levels defined by external benchmarks and frameworks of language ability. Implicit in this mapping is the identification of positive and negative writing features related to topic relevance, organization and structure, language and style. We consider these features in terms of the degree to which they cover models and constructs of writing ability. These features can be weighted in different ways to maximize their predictive power within different language use contexts and for different L1 backgrounds. Learners can therefore receive overall and specific feedback that diagnoses the quality of their writing according to context, increasing construct validity. Where use and impact are concerned, we have taken in to account research into how the intuitiveness, aesthetics and usability of diagnostic tools impact learning. In response, the system employs visualization techniques, highlighting parts of learners’ texts according to their overall quality. This is based upon a new innovative approach in computational linguistics weighting various factors from word and part-of-speech n-grams, identified grammatical constructions, and an estimated error rate. As writing quality improves, the background color gradually changes from red towards green. This displays in an intuitive way the transition from low-quality writing to high-quality writing. The system also stores all texts submitted by the learner. Within a learning-oriented assessment context learners can, perhaps assisted by their teacher, repeatedly access the system and continuously work on improving various aspects of their writing, referencing their earlier work as desired. Using questionnaires to collect feedback from teachers and learners, we provide empirical evidence that the system can be a useful supplement to other modes of learning that can help to promote learners’ writing development.