An Adaptive Learning Structural design for Next Generation Simulation Learning Systems

An Adaptive Learning Structural design for

Next Generation Simulation Learning Systems

In recent years there has been an increased use of serious games or immersive learning simulations to measure high-level learning processes and accelerate the transition toward expertise by motivating learners to acquire knowledge and skills by presenting instruction in dynamic, entertaining, and creative ways. As serious games have evolved, so has their design to accommodate user needs and simulate real world experience through high-fidelity visuals and interactions (Zielke, et al., 2009). Although user needs and hi-end visuals are important to user experience during game play, game design should concentrate on providing a contextualized, adaptive learning experience with detailed feedback given to a player based on his performance level. To address this issue, we have designed a theoretical framework that focuses on providing an adaptive learning experience whereby a user can progress or regress to appropriate skill levels in the simulation based on his response or non-response to well-crafted stimuli. The model is based on research examining how learners acquire new skills and make decisions in their natural settings under high stakes, time pressured conditions. In addition, we have applied this model as an enhancement to an existing government– owned, web-based platform that supports and trains decision-making at both operational and tactical levels. In 2008, the use of serious games as an effective training tool for learning and skill development of today’s workforce was predicted to reach 1.5 billion dollars in global market sales (Derryberry, 2007). Serious games are considered different from traditional video games because their intended objective is to educate military and corporate organizations with measurable and sustained outcomes in order to improve performance and behavior instead of just for recreation or personal use (Zyda, 2005). In this paper, we will describe several types of adaptive learning environments and present a theoretical design framework that illustrates user progression in a multi-tiered scenario-driven simulation environment. Also, we will discuss the components that constitute scenario-driven adaptive training systems, describe how the use of dynamic pathing and active sequencing in simulation design can allow players to assess and reflect upon their initial knowledge and skill level prior to and after game play, and then examine the criteria for advancement of performance levels. Finally, we will review a conceptual design of an After-Action Review that can show traceability of user performance in an adaptive simulation system and provide remedial training recommendations to the learner.

ADAPTIVE LEARNING ENVIRONMENTS

The emergence of adaptive learning environments for web-based education has been in existence since the early days of the internet (Brusilovsky, 1999). According to Brusilovsky, Pesin, and Zyryanov, (1993), adaptive learning environments are intelligent educational systems that combine features of traditional Intelligent Tutoring Systems (ITS) with hypermedia components which adapt and record learner performance as players interact with the system. Over the years, these system environments have manifested into multiple categories or types.

Brusilvosky & Paylo, (2003) and Specht & Burgos (2006) categorized adaptive learning environments into seven areas:

· Interface-based: This environment focuses on the user interface elements such as color, size, and shadow to support adaptive learning.

· Learning-centered: This environment focuses on the learning process by dynamically adapting the content to the instructional sequence the content in various ways. The learning path changes every time the learner starts.

· Content-based: This environment focuses on the presentation of content as it adapts to the learner’s experience. A content-based interface may include adaptive navigational support to guide learners towards relevant and interesting information. For example, content-based adaptive learning environments may provide direct guidance, adaptive link sorting/hiding/annotating/generation.

· Problem solving support: This environment guides the user on the appropriate steps in order to solve problems. The guidance could come from an intelligent mentor programmed by a predefined set of rules.

· Adaptive information filtering: This environment focuses on providing information that is relevant and sequenced by the performance of the learner.

· Adaptive evaluation: This environment focuses on the system or assessment content changing based on the performance of the learner to simulated injects and/or guidance of an intelligent mentor.

· Adaptive timing: This environment focuses on the ability for a system to modify or adapt a path based on the learner’s performance within a designated time allotment.

Types of Adaptive Feedback

Within adaptive learning environments, simulation feedback plays a vital role in assisting learner performance. Traditional simulation feedback allows a learner to reflect upon his performance after game play in order to think through the decisions and problems made during the simulation. However, in an adaptive learning environment simulation feedback may or may not assist a learner after an event while still in game mode to solve problems and realize workable solutions. During and after game play, feedback may take on many roles to inform, correct, and disrupt learner performance. For example, a mentoring agent could provide critical facts about the situation in order to inform a player. However, if a player disregards or does not have time to read the information, he may not possess the data needed to make an informed decision, thus potentially resulting in a negative consequence. The second type of adaptive feedback is corrective. Corrective feedback presents information in order to help improve or re-adjust a player’s performance. For instance, if a player is continuously making bad decisions, a mentoring agent might take over and pause the game to have the player participate in a remediation assessment in order to teach the proper knowledge and skills needed to continue with game play.

Another example of corrective feedback could be a simulated inject designed to scold a player for not taking the proper action or inaction. The third and final type of adaptive feedback is disruptive. The purpose of disruptive feedback is to confuse, interrupt, or lure a player from taking action on a specific task or a set of tasks. For example, a player is tasked to write an email warning of a pending disaster to a specific simulated character. As soon as the player begins to write the email, he is disrupted by a telephone call, then an important email message, then a video, and so on. The disruptive feedback could be one or a series of lures to distract the player for accomplishing the task or objective.

ELEMENTS OF SCENARIO DRIVEN ADAPTIVE TRAINING SYTEMS

For the purposes of our adaptive framework discussed in this paper and the model of the technology platform used, there are several design elements which must be acted on in order to develop an instructionally sound adaptive learning experience. The narrative, timeline, and injects are critical design elements which drive immersive learning experiences. The first design element is the narrative. The narrative is the storyline which chronicles past and future events and provides context to the learner during game play. In order to fully engage learners, it is important that the narrative design account for immersion. For immersion to take place within a narrative, the words must express a captivating story with a dilemma or problem in which players must engage and make decisions in the situation. Additionally, the narrative should start off by presenting general information of an event and provide details about the environment. As the situation unfolds, information and events reveal greater information, which enables learners to construct meaning and begin to make sense of the story. Real-world situations place severe time-pressure on decision makers, and so should simulated events. Adding disruptions or information overflow can increase the feeling of time-pressure. However, if too much time pressure exists, learners may make only quick reactions that prevent them from assessing situations and thinking through responses. Another element used in our theoretical adaptive framework is the development of multiple timelines along which a learner can progress through the game. Timelines provide an instructional designer with a structure around which he can organize detailed simulated injects. Every major event poses an opportunity for learners to encounter new challenges in which they must make decision(s) and take action(s) which will affect exercise outcomes. The third and final design element to make up a scenario-driven adaptive training system are simulated injects. Injects are the driving components of serious gaming simulations, allowing learners to take action and perform tasks. Once an instructional designer establishes the narrative and timelines, he begins to create injects which adhere to the narrative and timeline. Several elements are important to consider when developing injects for adaptive serious gaming environments: Ø Content: body of information conveyed to the learner Ø

Type: inject form (phone call, live conversation, video, 3D interaction, etc.) Ø

Time: inject occurrence on the timeline Ø

Simulated Entities: characters to and from whom the information is conveyed

Feedback: standardized responses to the injects/stimuli Ø Consequence: positive and negative outcomes of responding to an inject

 Advancement Criteria: measures used to indicate whether or not a learner progresses to higher level challenges. Criteria may take the form of a correct or incorrect response, time used to take action or inaction, etc.

Realistic, well designed injects motivate learners to engage in game play to make decisions and take actions. For instance, if the learner is presented with a live conversation inject and dialogue content is too ambiguous or does not use proper domain verbiage, the learner may or may not respond to the stimulus, thus compromising its original intent. Associating negative or positive consequences to simulated injects is also important. For example, in our adaptive learning model if the learner does not perform or respond to a task and there are no repercussions (e.g. make a follow-up telephone call) then the learner will fail to recognize the purpose of the task, may or may not be demoted to lower performance level, and will not complete the overall task objective. Injects, along with their associated actions, should allow learners to gradually gain insight into the nature of the scenario (Davis & Kahn, 2007). Injects should provide learners with the opportunity to expand their experience base and ability to structure information as it is collected. In addition, well developed injects allow learners to focus and comprehend the situation and enable them to assess events and reflect on actions.

A CONCEPTUAL MODEL OF AN INTEGRATED ADAPTIVE LEARNING ENVIRONMENT

The Importance Pre and Post-Testing

Pre- and post-testing can provide a measurement device for learners to assess their initial knowledge and skill levels in a particular domain or subject area and acts as a baseline assessment for them to select their appropriate skill level prior to game play. Pre and post assessments help an individual to practice the mission and think through potential problems and possible solution strategies. Additionally, preassessments can help activate already learned knowledge causing the player to have a potential advantage during game play because his mind is focused and primed on the tasks awaiting them. Post assessments help learners retain the knowledge and skills learned during the game by providing an opportunity for rehearsal. The more learners rehearse and practice with the appropriate instructional technique, the greater the likelihood they will be able to transfer the knowledge and skills into long-term memory (Clark, 2008).

Dynamic Pathing/ Active Sequencing

Dynamic pathing is the concept of taking different paths through a simulation based on the decisions made and actions taken within a defined time frame or what we call a “time horizon”. The decisions and actions selected by the learner are evaluated by the simulation portal in order to provide a customized learning experience tailored to his performance levels and abilities. Prior to game play the user selects an initial skill level or entry point into the simulation and then proceeds to play the immersive learning simulation (see Figure 1).

At the end of the first time horizon or timeline segment, the system analyses user performance based on the specific actions taken and decisions made in order to determine if the user should remain on the current timeline or be shifted to a alternate timeline based on his performance. In this way, the user’s experience is dynamically customized to provide an appropriate amount of challenge while playing the simulation. Figure 2 depicts a user who began playing the simulation on the “Novice” timeline and was shifted up to the “Intermediate” timeline at the end of “Time Horizon 1” based on good decisions and actions to simulation injects/stimuli that were presented within the timeline segment. Later in the simulation, difficulty levels for the learner were then subsequently increased further to the “Advanced” timeline and then back down to the “Intermediate” timeline at time horizons 2 and 3 respectively. This path through the simulation depicts a user who performed well and was promoted to more challenging timelines within the simulation. However, after reaching the “Advanced” timeline, the learner’s performance began to degrade during time horizon 3 and he was moved back down to the “Intermediate” timeline for the duration of the simulation (see Figure 2).

The concept of dynamic pathing can deliver many potential user experiences depending on what skill level is initially selected and what timeline paths the user follows through the simulation based on user performance. The notion of Active Sequencing provides for further customization of the learning experience within a timeline segment based on user performance. Based on the accuracy of decisions and/or actions made by the user, certain injects may be triggered or suppressed from being spawned within a time horizon which dynamically changing the user’s simulation experience. This feature helps the learner experience the consequences of his actions and decisions. By making good decisions, the learner can stave off potentially bad things from happening within the scenario. Conversely, making poor decisions or failing to take action can cause bad things to happen within the simulation similar to what happens in real life. Theoretical Design Approach Our theoretical adaptive framework is based on a model of performance and skill acquisition developed by Dreyfus and Dreyfus (1986). Their model describes a five-stage approach for skill and performance development ranging from novice to expert. Currently, the government-owned simulation platform used in conjunction with this conceptual model delivers game play at three varying levels of difficulty ranging from novice, intermediate, and advanced. These levels are selected by the learner at the beginning of the simulation.

Our enhancement to the existing serious gaming platform is to continue to give learners the option to select their initial competency levels, while allowing the simulation to adjust the difficulty level as the game progresses based upon the performance of its players. There are many games which currently offer their players a variety of novice, intermediate, and advanced modes of play.

Some, like Tetris, require an advancing player to perform essentially the same function at each stage, simply placing ever greater constraints on the time allotted to making decisions. Others, such as most racing games, present players with increasingly skilled opponents and require them to refine the subtleties of their own performance in order to achieve their objectives, while gradually limiting the margin for error. Games of this sort continually adapt in order to keep a player challenged, and can be highly effective for training a person to carry out specific tasks. Games in this vain are not, however, very useful instructional tools for teaching people how to effectively acquire and act upon information gathered from a variety of sources. To this point, few recreational nor serious games have yet been presented with environments which adapt to the ability of a player to manage dynamic and changing situations requiring more than the fulfilment of a single task.

A serious game which teaches its players to manage complex situations cannot adapt itself simply in speed of decision making or by limiting the margin for error. On the contrary, we propose that an effective training tool would in fact increase the possible margin for error by a rather large extent. The fundamental suggestion we are putting forth is that an integrated adaptive learning environment of this type is most effective when it offers its players an increasing level of freedom in choosing which actions to take and when to take them in response to the stimuli presented. Imagine a game in which a player has an ultimate objective at the end of a narrative, with a number of minor objectives set up along the timeline between the start and the finish, the resolution of which all have an effect on the success of the ultimate objective, but none of which can be singularly responsible for its success or failure. A novice player would be prompted by simulated entities with specific information and a choice of several pre-determined actions possible in response at set points within the timeline. He does not have a choice in when he receives the information, its source, or when to take action. His only freedom is to choose between several possible actions.

This format is actually employed already by a large number of games in order to familiarize the player with the interface. The idea to constrain the novice player’s freedom to make decisions is based on his limited experience and “context-free” rules in situations characteristic of their domain (Dreyfus and Dreyfus, 1986). Once the player has gained enough familiarity with the interface to know where information can be gathered and what sorts of actions are available within the simulated environment, however, this sort of guidance or mentoring should no longer be needed. At this point, a player reaches an intermediate stage. Obviously there can be several different levels of intermediate skill which a game can address. It is not important how many intermediate levels are available. The significant distinction between a novice and an intermediate player in the type of serious game we envision, however, is the freedom afforded the intermediate player in seeking information and choosing to act upon it.

 An intermediate player could be prompted with information by a simulated entity, for example, but not given any list of possible actions to take. He would have to decide upon his own action within the constraints of the simulated environment.

At another point within the narrative, a player could be presented with a minor objective, and a specific set of actions which could resolve the matter, but no information regarding which option would be most appropriate. It would fall upon the player himself to determine which of the different simulated entities within the game would provide the most useful information for achieving the objective. He could also be prompted with all of the information necessary to take a specific action, but given freedom to implement the action at a time of his own choosing, considering such variables as the changing resources available to him. Essentially the intermediate player is given a less rigid framework for achieving an objective than a novice player, but does not have to be completely self-reliant.

Also, he can choose where to find information, or what action to take, or when to take it, but not all at the same time. Because experts seem to have a highly tuned intuition that enables them to perceive situations and respond quickly and accurately (Klein, 1998), the advanced player, on the other hand, would be given wide freedom of choice in all of these matters. When an objective is presented, the player is responsible for consulting the appropriate simulated entities on his own in order to find out everything he needs to know in order to be able to choose a suitable action to take at an appropriate time. As much freedom as possible will be given to the player in order to accomplish each objective based on a variety of ways and varying degrees of success.

The potential for advancement between the levels will be determined by an assessment of each minor objective within the timeline. A novice should be advanced to the (first) intermediate level simply by completing his first objective. Effort should be made by the creators of the game to provide the player with as much exposure to the interface as possible within the framework of the first objective in order to assure a smooth transition. At the intermediate level, however, advancement should be granted only for a high degree of success in accomplishing a specific objective. This success can be measured in a variety of ways. A player who achieves a lower degree of success in completing an objective should be dropped down a level, where the increased level of prompting by simulated entities and constraint in available actions should help him focus more clearly upon his specific objective.

Furthermore, we suggest that a player be given the option to switch between his current level and novice level at any point, in case he needs the mentoring or interface clarification supplied there. Assessment of a player’s performance at the intermediate and advanced stages requires first that an objective be broken down into constituent components, such as information gathering, type of action, time of action, etc. If, for example, the game prompts the player with an objective, and the designers of the game have written the narrative such that there are four pieces of information held by the simulated entities which would be useful to the player in making a decision about how to act, a player would ideally take an action immediately after receiving the fourth piece of information. Few players could be expected to act so precisely, however. If information gathering were plotted on a bell curve, with four pieces of information at the apex, then a player who took action after receiving the fourth piece would be awarded a maximum score.

 If he acts too soon, with only three pieces of information, his score will be slightly less, but not significantly so, and when the objective is completed, the game will prompt him with feedback including the piece of information which was missed and how it applied to the decision to act. If the player continues to seek information after receiving the fourth piece, perhaps not realizing that he already possesses all of the knowledge needed to take responsible action, the assessment will account for each additional attempt to acquire information from simulated entities.

Prompting one or two entities for information will only have a slight effect on his overall score, but continued inquiry will have increasingly negative effects as the slope of the bell curve becomes steeper. Similarly, if he acts before acquiring much information at all, his score will be more highly impacted than if he had missed only one relevant piece of information. It is a simple matter to include in the assessment of each objective a clear explanation of why each of these pieces of information is important within the context of the objective.

 If the variable is time, an in-game clock could be started at a specific point along a timeline. The player is expected to take an action at a time of his choosing between the start and the stop of the clock. An ideal time for action must be chosen by the game designers. A bell curve could be drawn over that period of time. If the player acts too soon, or too late past the ideal time for action, he would receive a much lower score than if he acted within a standard deviation of the ideal moment. When the objective is completed, a less successful player could be prompted with specific reasons as to why he should have acted sooner or later than he chose to act, and how that choice affected the resolution of the objective.

If we imagine a situation which combined variables in both time and information gathering, it is possible that the player finds himself unable to acquire all of the relevant information by the time the ideal moment for action in the timeline arrives. If he takes action at this point, he should be given a full score for the time of the action, and a reduced score for the amount of information he had when he acted. The prompt after the resolution of the objective should explain to him that the time to act was appropriate, but reveal to him the significance of the information which he failed to acquire, and advise him where to locate similar pieces of information when encountered with future objectives in the same vain. The more advanced a player is, the more variables an adaptive learning simulation would present him with.

Figure 3 details a situation where a player has the freedom to choose between a number of possible actions, and to decide for himself the best time to take those actions. The combination of the three graphs shows how the bell-curve assessment strategy may be applied to similar situations, where freedom is afforded to a player within multiple variables in a simulation. The action assessment graph, on the left, is tilted ninety degrees from customary orientation for convenience. In this simulation, there are seven actions labelled A through G that are available to a player. Action D is theoretically the best solution to the scenario presented, and a player who chooses action D would receive the highest possible score from this graph for that choice. The time assessment graph shows that the best time for a player to take an action in order to resolve this scenario is at time signature 3.

Acting within a standard deviation of time signature 3 still has a profoundly positive effect on the resolution, but the further from time signature 3 that a player takes action along this timeline, the less effective his action will be in resolving the situation. Nevertheless, taking any sort of action in order to resolve the situation, even if grossly late, still has some value. It is for this reason that the bell curve is particularly suitable to these assessments, rather than a straight line. The graph showing the availability of actions over time, however, complicates the matter. Rarely, in a real world situation, are all actions available to a person at all times. Actions A and F, for example, are available essentially for the duration of the simulation. Action E, however, is available only at the beginning and end, and action D only after about time signature 4.8. As the simulation plays out, the actions available to a player at any particular moment can change. Theoretically, the best action to take and the best time to take it would be action D at time signature 3. This theoretical maximum is represented by the red dot. As one can see, in this particular simulation, a player does not have action D available at time signature 3, and is therefore forced to make a choice about taking less than-ideal actions at less-than-ideal times. The blue dot represents a relatively successful choice by a player, who takes a very good action, even if slightly early. The green dot represents a player who takes the exact same action, but far too late for it to resolve the scenario in an effective manner. The numbers next to each dot are the sum of the assessed actions and time signatures from their respective graphs. The assessment numbers in this specific graph, however, are quite arbitrary, and can be adjusted to fit any specific simulation.

Here, they are merely intended to illustrate a relative difference between possible choices. As one can see, the freedom afforded to an advanced player in such an environment greatly increases the margin for error from an ideal. For that reason, it is important that the feedback which a game provides to a player after the completion of an objective take these matters into consideration. The player represented by the blue dot should be praised for choosing a suitable action, even if it is not the ideal solution to the situation, because he acted in a timely manner. The player represented by the green dot, on the other hand, should not only be corrected for delaying his action for such a long time, but also for choosing a less-than-ideal action when the ideal action was available to him. A less advanced player presented with the exact same scenario would be limited with respect to one of the variables. He could, for example, be prompted by the game and told that he must take action E. He would then have the freedom to choose when to take that action, but nothing else. Conversely, a less advanced player could be prompted to take action at time signature 4. He would have the free choice between the five possible actions available at that time, but nothing else. A more advanced player, on the other hand, would be presented with the freedom of choice in increasingly greater numbers of variables, such as the gathering of information or the tapping of resources. No matter how many variables are included, however, assigning specific values to the choices available to a player, as we do here, makes it a simple task for the designer of a game to set limits above and below which the level of a game will automatically change. Perhaps our player represented by the blue dot will advance to the next level of difficulty because he scored above a 75, and our player represented by the green dot will regress for scoring below 55. The specific numbers are unimportant, and depend upon the discretion of the game developer. Bell curve assessments, however, are a simple way of quantifying the results of decisions made by players with respect to multiple variables.

CONCLUSION

The purpose of our theoretical adaptive framework discussed in this article is to demonstrate a blueprint toward developing an adaptive serious gaming system that could provide a greater opportunity for an organic learning experience whereby individuals can be consistently challenged in a meaningful way and granted freedoms to face challenges as they arise. This framework is meant to be a starting point for modelling, simulation, and educational professionals to continuously refine and advance as adaptive learning technologies are developed and implemented.

The emergence of taxonomies: looking at the past to understand the present

Information and knowledge have been classified for centuries. One of the first large organised catalogues was in the library in Alexandria in Egypt, during the 3rd to 1st centuries BC (Malafsky, 2008). Callimachus, its first bibliographer, compiled a 120-volume subject catalogue of all the library’s books, and is considered the founding father of librarians, as he not only listed the books but also included the author, data on the text and comments on authenticity to guide users and readers. In ancient Greece, Aristotle (384-322 BC) developed the first large biological catalogue, grouping animals with similar characteristics into genera and then distinguishing the species within the genera. 1 From Callimachus until the 1730s, classification, arrangement and taxonomy were all considered synonymous (Lambe, 2007). With time, the field of biology contributed to the development of taxonomy as an activity in its own right. This was a result of the work of two biologists with different ideas on how to order and arrange the rapidly growing knowledge base on species: George Louis Leclerc Compte de Buffon and Carl Linnaeus (ibid). Linnaeus was in favour of analysis and a controlled nomenclature of living organisms. De Buffon, on the other hand, advocated for an analysis of their environmental context. Linnaeus’s argument emerged as the winning one in the end. In his Systema Natura of 1735, he introduced a simpler way to distinguish species based on their anatomy. In his Critica Botanica of 1737, he proposed a binomial Latin naming system for different species. He adopted a hierarchical and nested tree structure to express genealogical relationships, which contributed to the development and acceptance of evolutionary theories of the late 18th century and early 19th century. Overall, this new system created a common language that greatly enhanced coordination and collaboration among botanists and biologists (Lambe, 2007). It simplified classification by imposing rigid rules and standardised approaches and is still used today in biology. The system has also influenced the management of information and knowledge in enterprises and organisations, as the next sections show. But what about de Buffon? While he may have lost the intellectual battle, his criticism of the system Linnaeus developed provides us with important lessons for the management of knowledge and information. De Buffon’s argument was that Linnaeus’s system did not capture the complexity of the biological. In his opinion, organisms could be classified in many different ways: by their environment, by their adaptations, by their functions, by their similarities in behaviours, etc.: the anatomy principle was only one way to classify creatures. By suggesting this multiple classification approach, he inadvertently created the basis for the faceted classification that is now applied widely in information systems. Lambe (2007) argues that the system Linnaeus developed highlighted the importance of simplicity and standardisation, which contributed to its wide acceptance. On the other hand, de Buffon’s greatest legacy relates to the many possible ways there are to organise the same things, with every arrangement telling a different story. The development of cataloguing systems continued to expand to different knowledge domains. One of the most influential classification systems has been the Dewey Decimal System, which was introduced in 1876 as the general catalogue system for libraries and which is still employed today (Hunter, ND; Malafsky and Newman, 2009).

Another famous taxonomy is the Bloom Taxonomy, which was first presented in 1956 by B.S. Bloom. This taxonomy is considered essential within the education community. It classifies different objectives that educators set for students (i.e. learning objectives): affective, psychomotor and cognitive. Within each, the higher level is dependent on a student having attained the knowledge and skills at lower levels (Bloom, 1956). The massive expansion of computing during the 1980s helped push information into digital formats, thus making it possible to distribute it on a much larger scale (Hedden, 2010). The advent of the internet in the 1990s then contributed to a further explosion of information dissemination and highlighted the need to develop new tools and skills to organise and retrieve such information (ibid). In this context, taxonomies have become necessary. But what is a taxonomy and how can it be defined? 

Progressive Organization of Subject Matter

ALLUSION HAS been made in passing a number of times to objective conditions involved in experience and to their function in promoting or failing to promote the enriched growth of further experience. By implication, these objective conditions, whether those of observation, of memory, of information procured from others, or of imagination, have been identified with the subject-matter of study and learning; or, speaking more generally, with the stuff of the course of study. Nothing, however, has been said explicitly so far about subject-matter as such. That topic will now be discussed. 

One consideration stands out clearly when education is conceived in terms of experience. Anything which can be called a study, whether arithmetic, history, geography, or one of the natural sciences, must be derived from materials which at the outset fall within the scope of ordinary life-experience. In this respect the newer education contrasts sharply with procedures which start with facts and truths that are outside the range of the experience of those thought, and which, therefore, have the problem of discovering ways and means of bringing them within experience. Undoubtedly one chief cause for the great success of newer methods in early elementary education has been its observance of the

contrary principle.

But finding the material for learning within experience is only the first step. The next step is the progressive development of what is already experienced into a fuller and richer and also more organized form, a form that gradually approximates that in which subject matter is presented to the skilled, mature person. That this change is possible without departing from the organic connection of education with experience is shown by the fact that this change takes place outside of the school and apart from formal education. The infant, for example, begins with an environment of objects that is very restricted in space and time. That environment steadily expands by the momentum inherent in experience itself without aid from scholastic instruction. As the infant learns to reach, creep, walk, and talk, the intrinsic subject-matter of its experience widens and deepens. It comes into connection with new objects and events, which call out new powers, while the exercise of these powers refines and enlarges the content of its experience. Life-space and life duration’s are expanded. The environment, the world of experience, constantly grows larger and, so to speak, thicker. The educator who receives the child at the end of this period has to find ways for doing consciously and deliberately what "nature" accomplishes in the earlier years.

It is hardly necessary to insist upon the first of the two conditions which have been specified. It is a cardinal precept of the newer school of education that the beginning of instruction shall be made with the experience learners already have; that this experience and the capacities that have been developed during its course provide the starting point for ah further learning. I am not so sure that the other condition, that of orderly development toward expansion and organization of subject-matter through growth of experience, receives as much attention. Yet the principle of continuity of educative experience requites that equal thought and attention be given to solution of this aspect of the educational problem. Undoubtedly this phase of the problem is more difficult than the other. Those who deal with the pre-school child, with the kindergarten child, and with the boy and girl of the early primary years do not have much difficulty in determining the range of past experience or in finding activities that connect in vital ways with it. With older children both factors of the problem offer increased difficulties to the educator. It is harder to find out the background of the experience of individuals and harder to find out just how the subject- matters already contained in that experience shall be directed so as to lead out to larger and better organized fields.

It is a mistake to suppose that the principle of the leading on of experience to something different is adequately satisfied simply by giving pupils some new experiences any more than it is by seeing to it that they have greater still and ease in dealing with things with which they are already familiar. It is also essential that the new objects and events be related intellectually to those of earlier experiences, and this means that there be some advance made in conscious articulation of facts and ideas. It thus becomes the office of the educator to select those things within the range of existing experience that have the promise and potentiality of presenting new problems which by stimulating new ways of observation and judgment will expand the area of further experience He must constantly regard what is already won not as a fixed possession but as an agency and instrumentality for opening new fields which make new demands upon existing powers of observation and of intelligent use of memory. 

Connectedness in growth must be his constant watchword. The educator more than the member of any other profession is concerned to have a long look ahead. The physician may feel his job done when he has restored a patient to health. He has undoubtedly the obligation of advising him bow to live so as to avoid similar troubles in the future. But, after all, the conduct of his life is his own affair, not the physician's; and what is more important for the present point is that as far as the physician does occupy himself with instruction and advice as to the future of his patient he takes upon himself the function of an educator. The lawyer is occupied with winning a suit for his client or getting the latter out of some complication into which he has got himself. If it goes beyond the case presented to him he too becomes an educator. The educator by the very nature of his work is obliged to see his present work in terms of what it accomplishes, or fails to accomplish, for a future whose objects are linked with those of the present.

Here, again, the problem for the progressive educator is more difficult than for the teacher in the traditional school. The latter had indeed to look ahead. But unless his personality and enthusiasm took him beyond the limits that hedged in the traditional school, he could content himself with thinking of the next examination period or the promotion to the next class. He could envisage the future in terms of factors that lay within the requirements of the school system as that conventionally existed. There is incumbent upon the teacher who links education and actual experience together a more serious and a harder business. He must be aware of the potentialities for leading students into new fields which belong to experiences already had, and must use this knowledge as his criterion for selection and arrangement of the conditions that influence their present experience.

Because the studies of the traditional school consisted of subject-matter that was selected and arranged on the basis of the judgment of adults as to what would be useful for the young sometime in the future, the material to be learned was settled upon outside the present life-experience of the learner. In consequence, it had to do with the past; it was such as had proved useful to men in past ages. By reaction to an opposite extreme, as unfortunate as it was probably natural under the circumstances, the sound idea that education should derive its materials from present experience and should enable the learner to cope with the problems of the present and future has often been converted into the idea that progressive schools can to a very large extent ignore the past. If the present could be cut off from the past, this conclusion would be sound. But the achievements of the past provide the only means at command for understanding the present. Just as the individual has to draw in memory upon his own past to understand the conditions in which he individually finds himself, so the issues and problems of present social life are in such intimate and direct connection with the past that students cannot be prepared to understand either these problems or the best way of dealing with them without delving into their roots in the past. In other words, the sound principle that the objectives of learning are in the future and its immediate materials are in present experience can be carried into effect only in the degree that present experience is stretched, as it were, backward. It can expand into the future only as it is also enlarged to take in the past.

If time permitted, discussion of the political and economic issues which the present generation will be compelled to face in the future would render this general statement definite and concrete. The nature of the issues cannot be understood save as we know how they came about. The institutions and customs that exist in the present and that give rise to present social ills and dislocations did not arise overnight. They have a long history behind them. Attempt to deal with them simply on the basis of what is obvious in the present is bound to result in adoption of superficial measures which in the end will only render existing problems more acute and more difficult to solve. Policies framed simply upon the ground of knowledge of the present cut off from the past is the counterpart of heedless carelessness in individual conduct. The way out of scholastic systems that made the past an end in itself is to make acquaintance with the past a means of understanding the present. Until this problem is worked out, the present clash of educational ideas and practices will continue. On the one hand, there win be reactionaries that claim that the main, if not the sole, business of education is transmission of the cultural heritage. On the other hand, there will be those who hold that we should ignore the past and deal only with the present and future.

That up to the present time the weakest point in progressive schools is in the matter of selection and organization of intellectual subject-matter is I think, inevitable under the circumstances. It is as inevitable as it is right and proper that they should break loose from the cut and dried material which formed the staple of the old education, In addition,  the field of experience is very wide and it varies in its contents from place to place and from time t, time. A single course of studies for ah progressive schools is out of the question; it would mean abandoning the fundamental principle of connection with life experiences.

Moreover, progressive schools are new. They have had hardly more than , generation in which to develop. A certain amount of uncertainty and of laxity in choice and organization of subject-matter is, therefore, what was to be expected. It is no ground for fundamental criticism or complaint. It is a ground for legitimate criticism, however, when the ongoing movement of progressive education fails to recognize that the problem of selection and organization ~ subject-matter for study and learning is fundamental. Improvisation that takes advantage of special occasions prevents teaching and learning from being stereotyped and dead. But the basic material of study cannot be picked up in a cursory manner. Occasions which are not and cannot be foreseen are bound to arise wherever there is intellectual freedom. They should be utilized. But there is a decided difference between using them in

the development of a continuing line of activity and trusting to them to provide the chief material of learning.

Unless a given experience leads out into a held previously unfamiliar no problems arise, while problems are the stimulus to thinking. That the conditions found in present experience should be used as sources of problems is a characteristic which differentiates education based upon experience from traditional education. For in the latter, problems were set from outside. Nonetheless, growth depends upon the presence of difficulty to be overcome by the exercise of intelligence. Once more, it is part of the educator's responsibility to see equally to two things: First, that the problem grows out of the conditions of the experience being had in the present and that it is within the range of the capacity of students; and, secondly, that it is such that it arouses in the learner an active quest for information and for production of new ideas. The new facts and new ideas thus obtained become the ground for further experiences in which new problems me presented. The process is a continuous spiral The inescapable linkage of the present with the past is a principle whose application is not restricted to a study of history. Take natural science, for example. Contemporary social Life is what it is in very large measure because of the results of application of physical science. The experience of every child

and youth, in the country and the city, is what it is in its present actuality became of appliances which utilize electricity, heat, and chemical processes. A child does not eat a meal that does not involve in its preparation and assimilation chemical and physiological principles. He does not read by artificial light or take a ride in a motor car or on a train without coming into contact with operations and processes which science has engendered.

It is a sound educational principle that students should be introduced to scientific subject-matter and be initiated into its facts and laws through acquaintance with everyday social applications. Adherence to this method is Mt only the most direct avenue to understanding of science itself but as the pupils grow more mature it is also the surest road to the understanding of the economic and industrial problems of present society. For they are the products to a very large extent of the application of science in production and distribution of commodities and services, while the latter processes are the most important factor in determining the present relations of human beings and social groups to one another. It is absurd, then, to argue that processes similar to those studied in laboratories and institutes of research are not a part of the daily life- experience of the young and hence do not come within the scope of education based upon experience. That the immature cannot study scientific facts and principles in the way in which mature experts study them goes without saying. But this fact, instead of exempting the educator from responsibility for using present experiences so that learners may gradually be led, through extraction of facts and laws, to experience of a scientific order, sets one of his main problems.

For if it is true that existing experience in detail and also on a wide scale is what it is because of the application of science, first, to processes of production and distribution of goods and services, and then to the relations which human beings sustain socially to one another, it is impossible to obtain an understanding of present social forces (without which they cannot be mastered and directed) apart from an education which leads learners into knowledge of the very same facts and principles which in their final organization constitute the sciences. Nor does the importance of the principle that learners should be led to acquaintance with scientific subject-matter cease with the insight thereby given into present social issues. The methods of science also point the way to the measures and policies by means of which a better social order can be brought into existence. The applications of science which have produced in large measure the social conditions which now exist do not exhaust the possible field of their application. For so far science has been applied more or less casually and under the influence of ends, such as private advantage and power, which are a heritage from the institutions of a pre-scientific age.

We are told almost daily and from many sources that it is impossible for human beings to direct their common life intelligently. We are told, on one hand, that the complexity of human relations, domestic and international, and on the other hand, the fact that human beings are so largely creatures of emotion and habit, make impossible large-scale social planning and direction by intelligence. This view would be more credible if any systematic effort, beginning with early education and carried on through the continuous study and learning of the young, had ever been undertaken with a view to making the method of intelligence, exemplified in science, supreme in education. There is nothing in

the inherent nature of habit that prevents intelligent method from becoming itself habitual; and there is nothing in the nature of emotion to prevent the development of intense emotional allegiance to the method.

The case of science is here employed as an illustration of progressive selection of subject-matter resident in present experience towards organization: an organization which is free, not externally imposed, because it is in accord with the growth of experience itself. The utilization of subject-matter found in the present life-experience of the learner towards science is perhaps the best illustration that can be found of the basic principle of using existing experience as the means of carrying learners on to a wider, more refined, and better organized environs world, physical and human, than is found in the experiences from which educative growth sets out. Some of the recent works in Mathematics for the Million, shows how mathematics, if it is treated as a mirror of civilization and as a main agency in its progress, can contribute to- the desired goal its surely as can the physical sciences. The underlying ideal in any case is that of progressive organization of knowledge. It is with reference to organization of knowledge that we are likely to find Either-Or philosophies most acutely active. In practice, if not in so many words, it is often held that since traditional education rested upon a conception of organization of knowledge that was almost completely contemptuous of living present experience, therefore education based upon living experience should be contemptuous of the organization of facts and ideas.

When a moment ago I called this organization an ideal, I meant, on the negative side, that the educator cannot start with knowledge already organized and proceed to lade it out in doses. But as an ideal the active process of organization facts and ideas is an ever present educational process. No experience is educative that does not tend both to knowledge of more facts and entertaining of more ideas and to a better, a more orderly, arrangement of them. It is not true that organization is a principle foreign to experience. Otherwise experience would be so dispersive as to be chaotic. The experience of young children centers about persons and the home. Disturbance of the normal order of

relationships in the family is now known by :' psychiatrists to be a fertile source of later  mental, and : emotional troubles--a fact which testifies to the reality of this kind of organization. One of the great advances in early school education, in the kindergarten and early grades, is that it preserves the social and human center of the organization of experience, instead of the older violent shift of the center of gravity. But one of the outstanding problems of education, as of music, is modulation. In the case of education, modulation means movement from a social and human center toward a more objective intellectual scheme of organization, always hearing in mind, however, that intellectual organization is not an end in itself but is the means by which social relations, distinctively human ties and bonds, may be understood and more intelligently ordered.

When education is based in theory and practice upon experience, it goes without saying that the organized subject-matter of the adult and the specialist cannot provide the starting point. Nevertheless, it represents the goal toward which education should continuously move. It is hardly necessary to say that one of the most fundamental principles of the scientific organization of knowledge is principle of cause-and-effect. The Hay in which this principle is grasped and formulated by the scientific specialist is certainly very different from the way in which can be approached in the experience of the young. But neither the relation nor grasp of its meaning is foreign to the experience of even the young child. When a child two or three years of age learns not to approach a flame too closely and yet to draw near enough a stove to get its warmth he is grasping and using the causal relation. There is no intelligent activity that does not conform to the requirements of the relation, and it is intelligent in the degree in which it is not only conformed to but consciously borne in mind.

In the earlier forms of experience the causal relation does not offer itself in the abstract but in the form of the relation of means employed to ends attained; of the relation of means and consequences. Growth in judgment and understanding is essentially growth in ability to form purposes and to select and arrange means for their realization. The most elementary experiences of the young are filled with cases of the means-consequence relation. There is not a meal cooked nor a source of illumination employed that does not exemplify this relation. The trouble with education is not the absence of situations in which the causal relation is exemplified in the relation of means and consequences. Failure to utilize the situations so as to lead the learner on to grasp the relation in the given cases of experience is, however, only too common. The logician gives the names "analysis and synthesis" to the operations by which means are selected and organized in relation to a purpose.

This principle determines the ultimate foundation for the utilization of activities in school. Nothing can be more absurd educationally than to make a plea for a variety of active occupations in the school while decrying the need for progressive organization of information and ideas. Intelligent activity is distinguished from aimless activity by the fact that it involves selection of means-analysis-out of the variety of conditions that are present, and their arrangement-synthesis-to reach an intended aim or purpose. That the more immature the learner is, the simpler must be the ends held in view and the more rudimentary the means employed, is obvious. But the principle of organization of activity in terms of some perception of the relation of consequences to means applies even with the very young. Otherwise an activity ceases to be educative because it is blind. With increased maturity, the problem of interrelation of means becomes more urgent. In the degree in which intelligent observation is transferred from the relation of means to ends to the more complex question of the relation of means to one another, the idea of cause and effect becomes prominent and explicit. The final justification of shops, kitchens, and so on in the school is not just that they afford opportunity for activity, but that they provide opportunity for the kind of activity or for the acquisition of mechanical skills which leads students to attend to the relation of means and ends, and then to consideration of the way things interact with one another to produce definite effects. It is the same in principle as the ground for laboratories in scientific research.

Unless the problem of intellectual organization can be worked out on the ground of experience, reaction is sure to occur toward externally imposed methods of organization. There pre signs of this reaction already in evidence. We are told that our schools, old and new, are failing in the main task. They do not develop, it is said, the capacity for critical discrimination and the ability to reason. The ability to think is smothered, we are told, by accumulation of miscellaneous ill-digested information, and by the attempt to acquire forms of skill which will be immediately useful in the business and commercial world.

We are told that these evils spring from the influence of science and from the magnification of present requirements at the expense of the tested cultural heritage from the past. It is argued that science and its method must be subordinated; that we must return to the logic of ultimate first principles expressed in the logic of Aristotle and St. Thomas, in order that the young may have sure anchorage in their intellectual and moral life, and not be at the mercy of every passing breeze that blows. If the method of science had ever been consistently and continuously applied throughout the day-by-day work of the school in all subjects, I should be more impressed by this emotional appeal than I am. I see at bottom but two alternatives between which education must choose if it is not to drift aimlessly. One of them is expressed by the attempt to induce educators to return to the intellectual methods and ideals that arose centuries before scientific method was developed. The appeal may be temporarily successful in a period when general insecurity, emotional and intellectual as well as economic, is rife. For under these conditions the desire to lean on fixed authority is active. Nevertheless, it is so out of touch with all the conditions of modern life that I believe it is folly to seek salvation in this direction. The other alternative is systematic utilization of scientific method as the pattern and ideal of intelligent exploration and exploitation of the potentialities inherent in experience.

The problem involved comes home with peculiar force to progressive schools. Failure to give constant attention to development of the intellectual content of experiences and to obtain ever-increasing organization of facts and ideas may in the end merely strengthen the tendency towards a reactionary return to intellectual and moral authoritarianism. The present is not the time nor place for acquisition upon scientific method. But certain features of it are so closely connected with any educational scheme based upon experience that they should be noted. In the first place, the experimental method of science attaches more importance, not less, to ideas as ideas than do other methods. There is no such thing as experiment in the scientific sense unless action is directed by some lead- idea. The fact that the ideas employed are hypotheses, not final truths, is the reason. Why ideas are more jealously guarded and tested in science than anywhere else. The moment they are taken to be first truths in themselves there ceases to be any reason for scrupulous examination of them. As fixed truths they must he accepted and that is the end of the matter. But as hypotheses, they must be continuously tested and revised, a requirement that demands they be accurately formulated.

In the second place, ideas or hypotheses are tested by the consequences, which they produce when they are acted upon. This fact means that the consequences of action must be carefully and discriminatingly observed Activity that is not checked by observation of what follows from it may be temporarily enjoyed. But intellectually it leads nowhere. It does not provide knowledge about the situations in which action occurs nor does it lead to clarification and expansion of ideas. In the third place, the method of intelligence manifested in the experimental method demands keeping track of ideas, activities, and observed consequences. Keeping track is a matter of reflective review and summarizing, in which there is both discrimination and record of the significant features of a developing experience. To reflect is to look back over what has been done so as to extract the net meanings, which are the capital stock for intelligent dealing with further experiences. It is the heart of intellectual organization and of the disciplined mind.

I have been forced to speak in general and often abstract language. But what has been said is organically connected with the requirement that experiences in order to be educative must lead out into an expanding world of subject-matter, 1 subject-matter of facts or information and of ideas. This condition is satisfied only as the educator views teaching and learning as a continuous process of reconstruction of experience. This condition in turn can be satisfied only as the educator has a long look ahead, and views every present experience as a moving force in influencing what future experiences will be. I am aware that the emphasis I have placed upon scientific method may be misleading, for it may result only in calling up the special technique of laboratory research as that is conducted by specialists. But the meaning of the emphasis placed upon scientific method has little to do with specialized techniques. It means that scientific method is the only authentic means at our command for getting at the significance of our everyday experiences of the world in which we live. It means that scientific method provides a working pattern of the way in which and the conditions under which experiences are used to lead ever onward and outward. Adaptation of the method to individuals of various degrees of maturity is a problem for the educator, and the constant factors in the problem are the formation of ideas, acting upon ideas, observation of the conditions, which result, and organization of facts and ideas for future use. Neither the ideas, nor the activities, nor the observations, the organization are the same for a person six years old as they are for one twelve or eighteen years old, to say nothing of the adult scientist. But at every level there is an expanding development of experience if experience is educative in effect. Consequently, whatever the level of experience, we have no choice but either to operate in accord with the pattern it provides or else to neglect the place of intelligence in the development and control of a living and moving experience.

Knowing What You Know: Assessment Competence

Learning is any process that enables a change in a person’s capacity to understand themselves and the world around them. Assessment is any feedback that enables a learner to know when there has been a change in their capacity to understand themselves and the world around them. Learning and assessment thus occur in the course of daily life as individuals interact with their environments, with each other, and as they internalize new knowledge and new ways of understanding. Assessment is integral to learning, whether provided in a formal classroom setting or any other context.

But learning is also an intentional process and competences for “learning to learn” are considered vital for effective lifelong learning.1 Learning-to-learn involves awareness of how and why an individual acquires and processes different types of knowledge (meta-cognition), and the learning methods, environments and habits she or he finds to be most effective (self-regulation) (Eurydice, 2002). Assessment is integral to all forms learning, whether the apparently effortless learning that occurs through interactions in daily life, or through intentional learning.

Formal assessments are based on quantitative and/or qualitative measurements. For novice learners, assessments are usually based on progress toward specific goals with clear criteria. At higher levels, learners may be working individually or collectively to solve problems with no clearly defined outcome, and assessment is part of this exploratory process. Feedback is an essential element in assessment to improve learning. Feedback may be provided in the course of an interactive discussion, in writing, in the context of a game, or through signals in the learner’s environment. It is most effective when it is timely and at an appropriate level of detail to suggest next steps (Black and Wiliam, 1998). In the case of goal-directed learning, feedback needs to be congruent with the learning aims. When learning is exploratory and open-ended feedback may, in some cases, be adequately conveyed through conventional and familiar indicators or it may not. In the latter case inventing new indicators and becoming sensitive to the meaning of such feedback becomes one of the key parts of improving assessment.

As already noted the context and technologies of learning and assessment are changing rapidly. Part of this change calls into question how we define and achieve the competences for learning to learn. Indeed, the term “competence”, which refers to the ability to mobilize both cognitive and non-cognitive resources in unknown contexts, seems particularly apt here. Learning, in both open and closed systems, requires learners to define what, how, with whom and why they learn; they and their collaborators, mentors and teachers need to extend and deepen their knowledge of effective assessment processes, and their ability to interpret and respond to feedback. Today’s changing context is calling for a deeper understanding of the factors (rational and non-rational) that enhance the ability of learners in all situations and environments to identify and track changes in their “capacity to understand themselves and the world around them”. This is assessment competency. 

New technologies are changing learning and assessment – and raising learners’ expectations that they will have a say in what and how they learn. They may also support new approaches to learning and education:

• Social media facilitate information sharing, usergenerated content, and collaboration in virtual communities. Increasingly, individuals expect the opportunity to participate in knowledge creation and to share their own assessments through blogs, wikis, feedback and rating services. Those with special interests may engage in collective learning and problem solving on a collective (crowd sourced) scale. In the education world, Open Educational Resources allow educators to share and shape learning content, and to provide input on effectiveness.

• Tracking programmes and electronic portfolios allow individuals to monitor their performance toward goals

(criterion-referenced assessments), to compare their progress against their own prior performances (ipsativereferenced assessments), or against the progress of other users (norm-referenced assessments). For example, Baumeister and Tierney (2011) note that several online programmes that help individuals to track exercise, sleep and savings habits can be very effective in supporting goal achievement. The research suggests that individuals who use tracking programmes to focus on how much further  they have to go (rather than progress already made) (Fishbach and Koo, 2010), and who share their progress with others are more likely to succeed in reaching goals.

• Video games both entertain players and provide rapid feedback on progress. Many games step up challenges as players develop their skills. Educationalists are now working with game developers to create games focused on building specific competences. For example, Popoviæ describes how novice learners in biochemistry have engaged in collective game play to address specific puzzles and problems within the discipline, and in the process have found solutions that have eluded experts. Delacruz and colleagues (2010) have found that effectively-designed games can tap into mathematical thinking and can be used for summative assessments of learning, and potentially, as more data on players’ thinking processes are gathered through game play, as a tool for formative assessment.

• Test developers are also focusing on how ICT-based assessments may support the integration of large-scale, external summative assessments and classroom-based formative assessments. Currently, there are a number of technical barriers to this kind of integration. Data gathered in large-scale assessments do not provide the level of detail needed to diagnose individual student learning needs, nor are they delivered in a timely enough manner to have an impact on the learning of students tested. There are also challenges related to creating reliable measures of higher-order skills, such as problem solving and collaboration.

• Learning Analytics, as discussed by Siemens in the following section, and by Shum in his work on Social Learning Analytics, take these ambitions a step further. In a previous paper, Siemens and Long (2011) noted

that social network analysis tracking learners’ online behaviour could potentially provide detailed information (“big data”) on learning processes and real time assessment of progress, along with suggestions for next steps. Importantly, learning analytics may also support “intelligent curriculum” and “adaptive content”, allowing deep personalisation of learning. Shum and Crick (forthcoming 2012) propose learning analytics as a tool to assess learners’ dispositions, values, attitudes and skills. These different technologies can facilitate assessment by and for learners. At the same time, they demand that both educators and learners develop deeper and more extensive competences for assessment.

Knowledge cannot be handled like a readymade tool that can be used without studying its nature. Knowing about knowledge should figure as a primary requirement to prepare the mind to confront the constant threat of error and illusion that parasitize the human mind. It is a question of arming minds in the vital combat for lucidity (p. 1) Morin, 1999. Knowing what you know, as Morin suggests, requires an understanding of the cultural, intellectual and cognitive properties and processes that shape knowledge, and the extent to which everything we know is subject to error, illusion and emergence of novelty. Assessment competences require an awareness of these vulnerabilities, and strategies to ensure greater capacity to define knowledge as it emerges from learning processes. The following elements are proposed as a framework for the development of assessment competences, with the hope of becoming better able to grasp emergence and to guard, as much as possible, against error and illusion.

1) Recognising the potential and limits of new and emerging assessment tools

Tools, whether high tech or as simple as a checklist or rubric, are an essential element of assessment. Many new and emerging technologies support greater user engagement in assessment processes, provide the timely

and detailed data needed to diagnose learning needs and connect users to relevant resources, build on effective tracking and monitoring strategies to support persistence toward goals, and scaffold learning challenges. Yet there are also some caveats to keep in mind. The quality of new and emerging ICT-based tools depends largely upon the quality of their design as well as the underlying algorithms. High quality tools will require significant investments in measurement technologies that take into account how people actually learn. Indeed, while cognitive scientists have made a great deal of progress in understanding the processes of learning in different subject domains over the last two decades, progress in measurement technologies has been much slower. Within the learning sciences, there is deeper understanding of how learners: move from novice to expert, develop typical misconceptions, create effective learning environments, and use self-assessment and meta-cognitive monitoring (Bransford et al., 1999; Pellegrino et al., 1999). Emerging technologies, including learning analytics, have ambitions of taking into account these advances in our understanding of learning as a cognitive and social process. However efforts to link technologies to effective

learning and assessment are at the early stages. Ensuring the validity and reliability of assessment tools calls for careful testing and innovative experimentation.

2) Making assessment criteria explicit

While technologies such as blogs, social media and rating services provide opportunities for anyone to express his or her opinion on any number of subjects, the criteria underlying assessments are not always made explicit. Both educators and learners need to be able to clearly identify criteria they are using to make judgments. For example, learners writing blogs need to learn to outline carefully constructed arguments and describe the basis on which they have developed an opinion or made a judgment – a skill emphasized in traditional schooling, but not always applied to the Internet. As consumers of information, learners also need to be able to identify the implicit basis for the evaluator’s assessment when criteria are not clear, and to appraise the validity of their judgments. Are ratings and assessments measuring what they purport to measure? How do others’ assessments relate to the learner’s own values and priorities and can online ratings inform their choices?

3) Strengthening meta-cognition and self-regulation

Meta-cognition (thinking about thinking) and selfregulation (self control) are both central to effective learning and assessment. This includes the way we construct our identities as learners, awareness of effective learning approaches and strategies, and how cognition affects perception and judgment, including the potential for bias and error. Self-regulation places the emphasis on the process of learning rather than the outcome. Assessment may focus on self-monitoring and the development of strategies to support persistence and effort. Here, the kind of tracking programmes mentioned above, that support self-assessment – whether against one’s own prior performances, against clear goals, or against the performance of peers (respectively ipsative-, criterionor norm-referenced assessments) may support the development of effective self-regulation.

Meta-cognition is also vital in understanding the quality of learning, whether toward specific goals or in exploring new ideas. Awareness of the potential for bias and error in assessments, are an important aspect of meta-cognition. Kahneman (2011) notes that, for the most part, we rely upon impressions and feelings and are confident in our intuitive judgments. Such judgments and actions may be appropriate most of the time. However, cognition can also induce people to make errors and biased judgments on a systematic basis. While individuals are capable of  sophisticated thinking, it requires much greater level of mental effort, and the normal human tendency is to revert to intuitive approaches. Awareness of these weaknesses and that they may be particularly dangerous in certain circumstances is a first step toward more effective assessment. More advanced assessment competences mean that individuals are better able to regulate the use of cognitive and intuitive approaches as well as recognize situations when error and bias might be more likely.

4) Judging the quality of information gathered in the assessment process

The quality of information gathered in assessment processes is vital for learning, whether outcomes are known or yet to be discovered. In the case of learning aimed at specific goals or outcomes, assessments of learner attainment are effective only if they are both valid and reliable. Validity means that the assessments measure what they purport to measure, while reliability means the assessment can be repeated and produce consistent results. Validity and reliability are important whether referring to large-scale, high-stakes assessments, classroom questioning, or self- and peer-assessment. The quality of information gathered in assessment is also vital for learning when it involves radical re-framing of ideas to open new ways of thinking and acting and finding novel solutions and tools to address problems. The learning outcome, in these cases, is unknown and assessment is more about testing ideas. It may involve “simple tinkering” or the development of predictive models to systematically test and verify hypotheses. Whether learning outcomes are known or not, both educators and learners need to develop an understanding of and appreciation for the quality of information gathered in different assessment processes. This includes an understanding of how different measurement models mediate that information, and recognition that no single assessment can capture all the information necessary. Multiple measures over time are necessary for more complete pictures. Assessment competences should thus include an understanding of the strengths and weakness of different approaches to measurement, and how they complement each other.

5) Inquiry

Both novice and expert learners may engage in inquiry as a form of deep learning. However, inquiry also requires strong competences to ask meaningful questions and to pursue lines of investigation. Questions, a form of assessment, provide a means to develop understanding, and to identify areas where thinking is unclear. For learners at “higher” levels of expertise, the outcomes of any inquiry may not necessarily be known. Thus, learners, collaborators, educators will need to develop competences to assess learning when information is incomplete and the answers are not clear. They will need to set the goals for different steps of the inquiry process, to develop criteria to gauge the quality of learning, and to develop ways to assess it. Here, the capacity to harness new learning analytical tools to mine “big data” and support inquiry hold particular promise.

6) Sense-making

Individuals, whether working individually or collaboratively, need to make sense of information – to weigh its value and to place it in context. Sensemaking competences include the ability to sort through information when there is too much of it, to recognize when there is too little information, to understand the ways in which individuals and groups shape information based on their own frames of reference, and similarly, the ways in which assessment technologies filter information. Sense making may also require reassessment of beliefs and prior knowledge in order to arrive at new and deeper understandings, and to create new meaning. Indeed, this kind of learning may be considered as “transformative” and can have a profound impact on development (Mezirow, 2011). The willingness to learn and to re-learn, to assess and to re-assess, to experiment and reflect creates the basis for wisdom (Miller et al., 2008).

7) Identifying next steps for learning and progress

When the aim of assessment is to improve future learning, then the information gathered in assessment processes should be linked to next steps for learning. The capacity to diagnose learning needs, and to identify appropriate strategies and resources for further learning, to search for further input from mentors and peers or the broader environment, are thus an essential part of the learning and assessment process. Indeed, this is an essential competence for learning-to-learn.

Assessment has always been an important competence for learning to learn, but it is increasingly central as new socio-economic contexts and new tools have increasingly required and enabled individuals to take charge of their own learning. The above elements represent a first effort at re-defining and deepening assessment competences as part of Morin’s ongoing “combat for lucidity” and knowledge in an emergent world.

Ultimately, assessment competences are about how each individual constructs her or his identity as a learner and as a person. Thus, the standards which guide a learners’ development as well as the ability to assess learning when there are no set standards, the capacity to re-assess and re-frame ideas and prior beliefs, and the ability to define next steps, are all vital steps along the path to lifelong learning.

Learning Analytics: A Platform for Learning

Education has long been a black box where educators and administrators have a limited understanding of the

methods, activities, and investments that produce quality learning. Learners in turn have only a limited awareness of the opportunities offered by tools and technologies to improve self-regulation in the learning process. Over the past decade, three trends have altered information spaces: quantity of data, type of data, and increased computation power. The quantity of data is evident everywhere, presenting challenges for individuals to cope with information overload. New types of data are produced through, and captured by, social media, mobile devices, and, increasingly, automated data capture through sensors (i.e. the Internet of things). Finally, as Moore’s Law has held reasonably firm, computation power has continued to grow exponentially.

As a result of these three trends, business and organizations increasingly recognize that data is an asset – a resource to be managed and leveraged. “Big data” has been coined to describe the growing quantity, and analytic opportunity, of data. PW Anderson, in 1972, prophetically anticipated the challenge facing organizations today: more is different. As data quantity increases, new approaches for mining and drawing insight from that data are needed. Businesses have responded to the new data reality through development of business intelligence. Governments and many organizations (such as OECD) have responded by opening up their data for researchers to analyze. In the education market, interest in learning analytics is growing as well. Learning analytics is the use of intelligent data, learner-produced data, and analysis models to discover patterns and connections within that data, and to predict and advise on learning.

At a basic level, learning analytics relies on some of the concepts employed in web analysis, such as Google Analytics, as well as those involved in data mining. These analytic approaches focus on learner activity through mining clicks, attention/focus “heat maps”, social network analysis, and recommender systems. Learning analytics extends these basic analytics models by focusing on curriculum mapping, personalization and adaptation, prediction, intervention, and competency determination.

Learners constantly put-off data – sometimes explicitly in the form of a tweet, Facebook update, logging into a learning management system, or creating a blogpost. At other times, unintentionally while in the course of daily affairs (or data that is provided by someone else – such as being tagged in Facebook). These data trails are captured by organizations, waiting for some type of analysis that generates insight of value. Amazon, Google, and Facebook use this data to personalize the experience for site users and to recommend additional resources or social connections.

Each new source of data amplifies the potential value of analytics. Most individuals have a fragmented digital profile with bits and pieces of data captured in LinkedIn, Facebook, Google, Twitter, blogs, and closed organizational systems. When these data sources are brought together, especially with physical world data captured by sensors or mobile devices, they can provide organizations with a profile of an individual (the integration of data sources was the primary intent behind the Information Awareness Office’s concept of Total Information Awareness following 9/11 in the USA). The recent development of semantic linked data holds promise in education. For example, “intelligent” data3 (machine-readable semantic or linked data) can 

be combined with learner-produced data, profile information, and curricular data for advanced analysis.

The data trails and profile, in relation to existing curricula, can be analyzed and then used to identify learners who are at risk of dropout or failure. This identification can then be used for automated or human intervention, personalization, and adaptation . Curriculum in schools and higher education is generally pre-planned. Designers create course content, interaction, and support resources well before any learner arrives for a course. This can be described as “efficient learner hypothesis”– the assertion that learners are at roughly

the same stage when they start a course and that they progress at roughly the same pace. Educators generally recognize that this is not they case. Learners enter courses with dramatically varied knowledge levels and progress through content at different paces. As a result, the process used by educational institutions in designing learning is in urgent need of restructuring.

Learning content, for instance, could be computed – a real-time rendering of learning resources and social suggestions based on the profile of a learner, her conceptual understanding of a subject, and her previous experience. Competence (as measured by a degree or certificate) need not even be explicitly pursued. For example, an integrated learning system could track physical and online interactions, analyze how learners are developing skills and competencies, and then compare formal and informal learning with discipline or field of knowledge. This comparison may be possible if a discipline has utilized intelligent/semantic/linked data to define its knowledge goals. Then, the learning system could inform a learner that (you are 64% of the way to a achieving a PhD in psychology, 92% to achieving a masters in science, 100% to achieving a certificate in online learning” and so on). If the learner then decided to pursue a PhD in psychology, the learning system could offer a personalized path forward that adapts constantly to knowledge the learner acquires in the course of work, formal learning, parenting, or the general process of “living life”.

Analytics for pre-defined goals can be approached from a bottom-up or top-down manner. Bottom up involves individual educators using single functionality tools to gain insight into the activities of learners. Fairly simple tools exist for conducting social network or discourse analysis. Additionally, statistics from a learning management system can provide data on login and posting frequency, indicators of learner specific habits. Top-down analytics, in contrast, require the formation of a strategy and vision on the part of an institution. When viewed from an institutional level, analytics can provide valuable information about learner success and failure by leveraging large datasets to reveal interesting correlations. Systems level analytics also gives an organization the ability to plan for automated interventions and develop learner profiles across various data sets. The analytics process, systemically deployed, consists of the stages.

When learning analytics are deployed systemically, several important, and integrated, functionality areas are needed . The analytics engine or module includes the specific analysis being performed on learner data such as social network analysis, predictions of learner failure, and evaluation of the impact of taking different course paths on learner success. The analytics engine draws data from learning management systems and personal learning environments, the social web, organizational learner profiles, and physical world data such as library usage.

The intervention engine is activated when learners exhibit signs of risk for dropout or failure or sub-optimal learning paths and activities. The adaptation and personalization engine uses intelligent learning content and compares learner ongoing activity with the goals or targets of a particular course or learning experience. Finally, dashboards enable end users (learners, educators, and administrators) to query data and visualize how different learners are performing. Many segments of the information industry – music, movies, news – have been dramatically altered by digital information and new options for end users to manage and control information. The financial and accountability pressures facing education, coupled with disruptive advances in technology, present a crisis point for the industry.

Education is adapting and evolving with the changes in today’s technology and information climate. Analytics hold the prospect of enabling changes and restructuring to the processes of teaching, learning, and administration. More importantly, analytics provide a feedback loop to track the impact of change initiatives, in education systems with predefined goals and potentially, as “assessment competences” improve, with open self-referential learning. Today’s recognition of societal complexity and the transformations occurring in the education market are providing important incentives for efforts to construct new platforms for improving the

quality of information that learners and educators use. Learning analytics open up the prospect of providing new resources for leaders, educators, and learners to make a leap in learning productivity happen.