\documentclass[12pt,a4]{article}
\usepackage{epsfig}
\usepackage[noconcise,finalfonts,header,report]{dfkititlepage}
\usepackage{isolatin1}
\scrollmode

%  \leftmargin-1cm
%  \textwidth15cm
%  \textheight22cm

\def\xml{{\tt XML}}
\def\html{{\tt HTML}}
\def\omdoc{{\tt OMDoc}}
\def\openmath{{\tt OpenMath}}
\def\mathweb{{\sc MathWeb}}
\def\activemath{\sc ActiveMath}
\def\mbase{{\sc MBase}}
\def\nextbest{suggestion mechanism}
\newcommand{\comment}[1]{} 
\newcommand{\ednote}[1]{} 

\title{Evaluators and Suggestion Mechanisms for {\activemath}}

\begin{document}
\reportnumber{RR-02-01}
\burstfkzitw{}
\bursttitle{Evaluators and Suggestion Mechanisms for {\activemath}}
 \author{Erica Melis  and Eric Andr\`es}
\date{}
 \maketitle
{\small \tableofcontents}
\newpage

\begin{abstract}
  This report describes the feedback in {\activemath}, a learning environment
  for mathematics. In particular, we distinguish local and global feedbacks.
%
  For the global feedback we describe the blackboard-architecture with its
  reusable and easily modifiable components, specify the current evaluators of
  the learners actions, and present the user-adaptive global learning
  suggestions implemented so far.
%
  The report also describes the design of the suggestion  user interface that 
is
   based on empirical results about feedback described in the
  literature. Finally, remaining questions that need to be clarified by
   empirical research are discussed.
\end{abstract}

%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}
%%%%%%%%%%%%%%%%%%%%%%%%
%in paper anderer Anfang
%role of feedabck in learning: direct in problem solving and delayed 
%activemath
{\activemath} is a user-adaptive web-based learning environment for 
mathematics.
It generates learning material for the individual learner according to her 
learning
goals, preferences, and mastery of concepts as well as to the choosen learning
scenario \cite{activemathAIEDJ01}. 

{\activemath}'s user model consists of the components history, profile, and
a database of mastery-level for the learning objects.
The history contains information about the user's activities (reading time for
items, exercise success rate, editing of user model).
The user profile contains her preferences, scenario, goals submitted for the
session.
To represent the concept mastery assessment, the  user model
contains values for a subset of the competences of Bloom's mastery taxonomy
\cite{Bloom}:
\begin{itemize}
\item Knowledge
\item Comprehension
\item Application. 
\end{itemize}
We assume that knowledge-mastery is reached by reading a text but also from
examples and exercises. Bloom defines the skills needed for 'knowing' as recall
of information, knowledge of major ideas; Comprehension-mastery can be achieved
by relating several concepts and then answer questions and understand examples.
Bloom defines the needed skills as understanding information, translating
knowledge into a new context, predicting consequences; Application-mastery
relates to solving problems by applying the concept. Bloom defines the needed
skills as usage of the concept (in new situations), solving problems using the
required skills or concepts.\footnote{Transfer-mastery could be the most
  advanced mastery but currently, we subsume this in application currently.}
Similar categorizations of mastery have been employed by several researchers,
e.g., Merrill and Shute \cite{ShuteSMART} in their taxonomies of
outcome types. 
%
Finishing an exercise or going to another page triggers an updating of the user
model in {\activemath}.  Different types of user actions influence the values 
of
corresponding competencies because each reflects a different competency
\footnote{The implicit connection between competencies is also considered but
  will be better formalized in Bayesian Net user model which is in progress.}.
In particular, reading concepts corresponds to 'Knowledge', following examples
corresponds to 'Comprehension', and solving exercises corresponds to
'Application'.

So far, {\activemath} primarily served the adaptive (static) document
generation, the integration of service systems, and local feedback in 
exercises.
This report addresses an extension and discusses  different types of feedback 
in {\activemath}.  We
suggest to distinguish local and global feedback. The global feedback is a new
service of {\activemath} which is described in this report. In particular, we
describe the architecture, the kinds of evaluators of student action, and some
of the user-adaptive global learning suggestions implemented so far.

%%%%%%%%%%%%%%%%%%%%%%%%
\section{Separation of Local and Global Feedback}\label{sec:globallocal}
%%%%%%%%%%%%%%%%%%%%%%%%
 Traditionally, user-adaptive feedback and help in tutor systems (ITSs) has
been designed for a direct response to students' problem solving actions and 
the
feedback is designed to help students to accomplish a solution, e.g., in the
PACT tutors \cite{PACTtutors97} or in CAPIT \cite{MayoMitrovichMcKenzie00}.
% 
Frequently, this feedback is an explicitly authored text that reflects the
author's experience with typical errors in a specific problem solving context.
In some sense, the specific feedback is questionable because authoring all the
specific feedback is a very laborious task (see e.g., \cite{VanLehn01}) and
often requires an extreme authoring effort for
explicitly authoring what can go wrong and what the reason is for each 
erroneous
action in each exercising. Therefore, we try to avoid such a kind of diagnosis
and corresponding feedback in {\activemath}.
%das passt hier eigentlichnicht so recht
Moreover, the usage and benefit of more and more detailed help may strongly
depend on the individual user \cite{AlevenKoedingerITS00} and some users might
even dislike frequent suggestions and intrusion \cite{ConatiAIED01}.

Although in most ITSs the feedback is a direct answer to single problem solving
steps, some systems provide feedback targeting meta-cognitive skills of the
student. For instance, \cite{ConatiAIED99,AlevenKoedingerCross99} try to 
support
self-explanation of worked-out examples; SciWise \cite{SciWise99} provides
feedback for planning and reflecting activities in experimental science and for
collaboration.
%MetaLink \cite{Metalink99} supports inquiry activities in hypermedia with
%implicit info(no feedback though).

Generally, two kinds of feedback and guidance can be provided by an ITS, a {\em
  local} response to student activities which is supposed to coach the
recognition of errors and the correction of single problem solving steps of the
learner and a {\em global} feedback targeting the entire learning process.  
This
differentiation somewhat resembles the distinction of task-level and high-level
described in the b4-process model of \cite{Mislevy99}.

These two kinds of feedback differ with respect to time, realm, content, and
aim. For instance, local feedback is provided immediately after each problem
solving action of the user and local feedback should be given directly attached
to the problem solving activity and possibly presented in the exercise window.
Instead, the global feedback and suggestions can be provided independently and
may be delayed, i.e. delivered, when the user has finished reading a text,
studying an example, or working on an exercise.

Many ITSs do not provide global feedback at all. And even if they do, such as
SQL-Tutor \cite{MitrovichOhlsson99} and CAPIT \cite{MayoMitrovichMcKenzie00},
they do not separate local and global feedback, say architecturally.
% but,  at most,  by using different  functions of the same module. 
% A separation might  be unnecessary because these systems  cannot use an 
external
% problem solver for producing the  local feedback -- as {\activemath} does, 
e.g.,
% with Computer  Algebra Systems.   

In {\activemath}, local and global feedback is distinguished because of
their different aims, different foci, different learning dimensions, and
different mechanisms.  In addition, the employment of service systems for the
check of the correctness of a problem solving step and for the generation of
local problem solving feedback \cite{its02} is a practical reason for 
separating local and
global feedback.  The local feedback such as 'syntax error', 'step not correct,
because...', 'task not finished yet', or 'step not applicable' is computed by a
system and related to a problem solving step in an exercise or to the final
achievement in an exercise.  The current implementation of local feedback is
explained in \cite{its02} and more technically in {\tt
  http://www.ags.uni-sb.de/\~{}adrianf/activemath/}.

As opposed to the local feedback, the global feedback scaffolds the student's
navigation, her meta-cognition, and dynamically suggests appropriate
presentations of content (including examples and exercises).  The global
feedback and suggestions may concern, e.g.,  what to learn, to repeat,
to look at, or to do next, navigating the content, reflecting, monitoring.
%was ist hier eigentlich Fehler finden??

The computation of global feedback requires diagnostic information of several
user activities. The information about the student's navigation, reading,
understanding, and problem solving actions, e.g. their duration, and success
rate, has to be used as a basis for user-adaptive suggestions. That is, 
information about the history of the learner's actions and information from her
user model is necessary to generate useful suggestions.

 In what follows we deal with global feedback only.

\paragraph{Notation.} \ednote{wir das gebraucht?}
  Note that K/C/A-present($c$) is
an abbreviation for: present content contributing to the concept $c$ in a K-,
C-, or A-oriented way respectively.
%
K/C/A-present($c$) are functions of the {\activemath}' existing course
generator.  K-oriented means present just concepts and possibly explanations;
C-oriented means present concepts and examples; A-oriented means present the
full spectrum of content including concepts, examples, and exercises.

%%%%%%%%%%%%%%%%%%%%%%%%
\section{Blackboard Architecture for Global Feedback}
\label{sec:global}%\label{sec:architecture}
%%%%%%%%%%%%%%%%%%%%%%%%
The architecture for the global suggestion mechanism  in {\activemath} clearly 
separates diagnoses
and suggestions as shown in in Figure \ref{fig:evalsugg}.  An advantage of the
separation of evaluation and suggestions is that the same evaluation results 
can
be used by different suggestion mechanisms, in different pedagogical 
strategies,
and later also by a dialog system.  For instance, if the diagnosis yields a 
{\tt
  seen(example, insufficient)}\footnote{i.e., the time for reading the example
  is less than a threshold}, then {\tt example} is presented again in a 
strict-guidance
strategy but not in a weak-guidance strategy.

\paragraph{Evaluators}
Some evaluators provide a diagnosis immediately from one of a user's action 
while
other evaluators infer a diagnosis from the immediate diagnoses and additional
information In Figure \ref{fig:evalsugg}, several {\em immediate} and one
{\em intermediate} evaluator are displayed.  New immediate and intermediate 
diagnosis
agents can be easily added, e.g., an evaluator for the individual average
reading time.

\begin{figure}[htbp]
   \begin{center}
 %    \strut\epsfxsize=9cm
\includegraphics[width=11cm]{/home/melis/schreiberei/Figures/feedbackArch}
%\includegraphics[width=11cm]{/opt/local/home/erica/schreiberei/Figures/feedbackArch}
     \caption{The architecture of evaluator and suggestion mechanisms}
     \label{fig:evalsugg}
   \end{center}
\end{figure}

The  immediate evaluators each watch one of the following types of activities
\begin{itemize}
\item navigation
\item reading (time)
\item problem solving (assessed performance)
  \begin{itemize}
  \item MCQ exercises
  \item exercises with a Computer Algebra System
   \item exercises with the Omega proof planner
  \end{itemize}
\end{itemize}
 
The immediate evaluators pass their results to a {\em diagnosis blackboard} 
(DBB)
and to the user model (user's mastery-level of concepts and to the activity
history). The current updating mechanism for mastery-values in the user model 
is
described in  \cite{activemathAIEDJ01}. Essentially, K-mastery values are
triggered by reading, C-mastery values by dealing with examples, and A-mastery
values by dealing with exercises.  A Bayesian Net user model including its
updating mechanism is future work.

{\em Intermediate} diagnoses
are computed by other evaluators from the information on the DBB and in the 
user
model. These diagnosis are  written on the DBB too.  
%

As displayed in Figure \ref{fig:evalsugg}, several suggestors compute global
feedback from the diagnoses on the DBB and write on the {\em suggestion
  blackboard} (SBB).   If necessary, the results are sent
to a ConflictManager that rates the different suggestions on the SBB. Then the 
best
rated suggestions are executed.
%%%%%%%%%%%%%%%%%%%%%%
\section{Evaluators/Diagnoses}
%%%%%%%%%%%%%%%%%%%%%%%%


Currently, {\activemath} has the following  evaluators, each one watching a 
particular type of activity
of the user or the DBB.
\begin{itemize}
\item navigation
\item reading 
\item exercises
  \begin{itemize}
  \item MCQ exercises
  \item exercises with a CAS
  \item exercises with the Omega proof planner
  \end{itemize}
\item intermediate evaluators
\end{itemize}

These evaluators return  diagnoses, among them
\begin{itemize}
\item mastery with values {\tt poor}  or {\tt okay}.
 Poor means that the success rating was below 30\%.
This evaluator acts when the user has finished an exercise.

\item {\tt navigation} with values {\tt okay/(irrational, ?start)}, where {\tt
    irrational} is diagnosed 
in case the user navigated from one concept to another without following any 
dependency.
   \comment{Berechnung vielleicht 
  diff(plannedHistory EA->Erica: Ist das nicht mit dem TOC unifizierbar?, actualHistory) und annotiere eine ?welche? History mit
  fehlenden K,C,A und eventuell irrationalNavigation ??}

%%read-resultate
\item 
{\tt focus-concept}(?c)
\item 
{\tt seen} with arguments ID and one of the values {\tt 
okay/notSeen/insufficient}
 for evaluated reading time of  an item ID. 

%%exercise-resultate
\item 
{\tt K,C,A-mastery} of a concept with values that correspond to a
percentage. This is  passed to
  the user model for updating and written onto the DBB. 
\item 
{\tt solution} with the arguments ID (of exercise) and one of the values {\tt 
correct/incorrect}.
\comment{spaeter mehr?}
\item %im Authorentool noch zu unterstuetzen
{\tt teacherDiagnosis(?x)}, where ?x is the name of a
typical error.
%
If the learner's input matches with a typical error, typicalError(?x), 
pre-defined
by the author of an exercise, the evaluator returns to the DBB a corresponding
tuple {\tt (teacherDiagnosis(?x), teacherFeedback(?x), teacherReaction(?x))}
that is pre-defined by the author.
%
  \comment{Eric Adrian: soll das in den local eval
    code??} If {\tt teacherDiagnosis(?x)} provides a mastery diagnosis, then 
it is  passed to the user model as well. 

\end{itemize}  

The following intermediate features can be inferred from information in the DBB
and in the user model:
\begin{itemize}
\item 
   {\tt missingPrerequisite(?concept, ?level)}, where
  {\tt ?level} can be K, C, or A currently.  This fact means that {\tt 
?concept} is a
  prerequisite concept of the focus-concept and its {\tt ?level}-mastery in the
  user model is insufficient.  This fact is inferred only in case of an
  insufficient result of the user actions related to the focus-concept.
  
 %  intermediate features: in-K-goals, in-C-goals, and in-A-goals which
%   provide the the set of concepts which have too low a K-, C-, or A-value
%   respectively. These are inferred from K,C,A-failing prerequisites of the
%   focus-concept in case of insufficient results of user actions on the focus 
%   concept
\comment{fuer Eric: evaluator berechnet aus geringer K,C,A-mastery fuer 
focus-concept $C$ oder aus
  solution(ID, incorrect) und user model mastery} \ednote{Carsten fragt: warum
  wollen wir den Benutzer nicht auf fehlendes Vorwissen aufmerksam machen, auch
  wenn er alles richtig gemacht hat? EA: Man sollte besser in so einem Fall 
eine
  Verstaendnisfrage stellen, da es ja sein kann, dass das Benutzermodell das 
Vorwissen
  falsch eingeschaetzt hat. EM: ja gut. noch nicht implementiert}

\item {\tt SeenAndKnown(?level)}, {\tt notSeenButKnown(?level)},{\tt SeenButUnknown(?level),
    notSeenAndUnknown(?level)}, all meant for the focus-concept, where {\tt
    ?level} is one of the values K, C, or A.  These intermediate features can 
be
  inferred from the {\tt focus-concept}, from the {\tt seen} diagnoses of the
  items contributing to the {\tt focus-concept}, and from the {\tt
    K,C,A-mastery} for the {\tt focus-concept}.  

  \comment{Carsten fragt: warum nur fuer focus concept? Was wenn mehrere
    concepts auf einer Seite? EM: eigentlich soll immer nur ein concept auf
    einer Seite bzw im focus sein. Jedenfalls sollte NextBest fuer jedes 
concept berechnet werden, oder?}
% \item  {\tt seenButNoImprovement(ID, ?n)}, where {\tt
%     ?n} is the number of times ($?n > 1$) the user has coped with the item
%   (definition, theorem, example, or exercise). It applies to an item
%   contributing to the focus-concept. 
% %
%   It can be inferred from the {\tt
%     focus-concept}, the direct {\tt seen} features of the items contributing 
to
%   the {\tt focus-concept}, from the {\tt K,C,A-mastery} for the {\tt
%     focus-concept}, and from the history information how often ID has been
%   visited by the user.

\end{itemize}

The following diagnostic features may  be added  later.
\begin{itemize}
\item self-guidedness\ednote{EA: is already added}
\item   average reading speed 
\item CAS-ability
\item Omega-ability
\item  the user's motivational state.

 % with the values littleMotivation,...\ednote{Renkl, Sabine?}
\item  
percentage of guess and/or slip \ednote{WIE LERNEN WIR DAS Eric:Training eines 
BN? IRT? EA: Muss man sich nochmal ueberlegen. Im Moment gar nicht.}
\end{itemize}

In the following, we describe the evaluators for reading,
navigation and for the intermediate  features in more detail. 
The
evaluators for exercises are directly implemented in the exercise environment
and pass their results to the DBB. 
%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Evaluators for Reading}
%%%%%%%%%%%%%%%%%%%%%%%%
The device the reading evaluator is based upon is a {\em poor man's eye
  tracker} \cite{eLearn2002}.  This is a software component that, based on 
mouse movement, can
approximately measure the time a user spends looking at a unit, i.e., a
definition, a motivation, a theorem, an exercise, or an example. 

A reading evaluator calculates its results from the actual reading time for a
unit returned by {\activemath}' poor man's eye tracker and from the {\tt
  averageReadingSpeed}.  The evaluator can provide the values okay, notSeen,
insufficient, and unclear of the feature {\tt seen}.    Note that the required 
threshold for a
sufficient reading time might vary between a definition, a theorem, an
explanation, an example, and an exercise. Note also that the reading speed for
formulas may individually differ from the reading time for text.\ednote{Erica: 
Das ist
  noch nicht implemetiert (different threshold). Man kann es aber recht leicht
  implementieren. Soll ich? EM-> Eric: ja, aber wir muessen vorher Information 
finden, was
  der Unterschied genauer sein koennte. Vielleicht koennen wir das aufgrund der
  logfiles finden, die in GG's Uebung anfallen werden. Schau in Conati's Diss,
  weas sie ueber average time schreibt. EA: OK...}

A read-evaluator updates the user model's K-value for a concept (i.e., the 
value for
knowledge-level mastery) after the user has looked at an instructional item that
contributes to learning the concept and C-values (for
comprehension-level mastery) after reading examples in particular.

\ednote{jetzt erst mal so lassen. Eric: was passiert, wenn user schon gute k bzw C-values hat und dann
  nicht mehr genügend lange liest? Das kann ja bedeuten, dass er es nicht mehr
  nötig hat, länger zu lesen (also Werte nicht heruntergesetzt werden müssen)
  oder dass er doch nicht alles mitbekommt (also Werte heruntergestzt werden
  müssten (Eric->EM: oder nur nicht hoch? runtergestzt wird gar nicht, bzw nur
  bei exercises). Da hilft eigentlich nur, mit NextBest Verständnisfragen zu stellen, um mehr
  Klarheit zu gewinnen. Und das wäre ein Fall für Diagnose UNCLEAR. 999}
\ednote{Erica, unkommentiere das wenn Du willst: The A-Value is also affected
  a bit, because  a student can already get an idea of how to apply the concept
  while reading it and while seeing examples. EM -> Lieber nicht schreiben, weil
  das updating doch mindestens nur sehr gering sein kann.}


%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Evaluators for Navigation}
%%%%%%%%%%%%%%%%%%%%%%%%
The navigation behaviour of a student has to be monitored because a user with
little experience may get lost in a (collection of) hypermedia document
\cite{MurrayEtal00}.  {\activemath}'s user tracking device, DFKeye
\cite{eLearn2002}, communicates to the reading evaluator the page
and the time the student spent on it. The evaluator 
diagnoses, whether the navigation behaviour makes sense according to 
\begin{itemize}
\item the position of the page relative to the previous one in the user's history
\item the relation of the focus concept on the current page with the focus
  concept of the preceeding one. 
\end{itemize} 
If the current page is immediately following or preceeding the previous page,
the navigation is rated okay. In this case, the user is either following the
curriculum that was proposed, or is going back to see the last concept that was
presented again.\ednote{Eric:muss noch bearbeitet werden!!!!!
 aber was wenn er mehrere Schritte zurueck
  geht. Das waere der Beschreibung zufolge auch okay, weil du ja immer relativ
  zur history nachsiehst und nicht relativ zum generierten Vorschlag.} 
\ednote{EA->EA: Koennte Sinn machen, dass manche
  navigationsschritte neutral sind, so wie eine Seite zurueckgehen, (EM: ja aber
  nur 1 Seite) waehrend
  eine Seite vorwaertsgehen positiv ist... (wenn das Konzept beherrscht ist?
  schwammig) EM: Dem generierten folgen (im static course oder in NextBest!!)
  ist doch eigentlich immer okay was die 
  Navigation angeht, auch wenn die mastery noch nicht vorliegt - aber dann gibt
  es ja NextBest Vorschlaege fuer concepts. Aber falls ein concept schon
  gemeistert wird, ist ein Auslassen der Seite auch okay.}
 If the new focus is in any relation with the old one (e.g., a
precondition), then the navigation is also okay.  These conditions 
define the {\em positive} navigation actions. If a navigational action is not
positive, we call it {\em negative}. \ednote{neutral waere auch ganz
  gut. EA->EM: Mein Punkt waere ja gerade , viellleicht 3 Arten zu unterscheiden (pos.,neutral,neg.). Wobei dann neutral auch unclear sein koennte. } The evaluator keeps track of both types of 
navigational actions. If the user seems to be lost, that is, she performs many
\ednote{Eric: was heisst many? -> Konfigurierbar, zur Zeit 3. Bis jetzt werden
  nur ANz der pos und neg NavSchritte aufsummiert. Hier weitermachen!!!!! und
  verschiedenes Feedback geben je nach Abweichungen.}
negative navigational actions, it diagnoses {\em irrational navigation} and 
passes this to the DBB. This can lead to the generation of a hint that helps the
user on her way through the hypertext.  Conversely, if the user's 
 navigational actions are rated positively, then it diagnoses {\em navigation okay} and this  can
cause the generation of reassuring feedback. 

\ednote{Eric:  self-guidedness ist aber nicht nur ein Navigation feature sondern
  ein LernMerkmal des Lerners (affective?) und koennte aus histories berechnet werden.}
A more complicated evaluation function is planned that results in the diagnose 
{\em self-guided navigation} for a navigation behaviour that does not fully
follow the sequencing suggested by the system (static book or dynamic
suggestions) but is in line with the user's mastery level. That is, if the user
discovers that she does not know enough about several  prerequisites and
navigates back to pages concerned with the prerequisite-concepts or -items and then
returns to the current page, this is taken as an evidence for her awareness of
the course structure and dependencies as well as for self-guidedness. 

\comment{
%%%%%%%%%%%%%%%%%%%%%%
\subsection{Evaluators for Exercise Activities}
%%%%%%%%%%%%%%%%%%%%%%%%
888
%%%%%%%%%%%%%%%%%%%%%%%%
%\subsubsection{Evaluators for Proof Planning Activities with Omega}
%%%%%%%%%%%%%%%%%%%%%%%
888

%%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Evaluators for CAS Activities}
%%%%%%%%%%%%%%%%%%%%%%%
888

%%%%%%%%%%%%%%%%%%%%%%%
\subsubsection{Evaluators for Multiple Choice Question Activities}
%%%%%%%%%%%%%%%%%%%%%%%
888
}
%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Evaluators for Intermediate Features}
%%%%%%%%%%%%%%%%%%%%%%
Each evaluator for intermediate diagnosis  watches the DBB-entries and the
user model and infers intermediate diagnoses. Currently, these are the 
following:
\begin{itemize}
\item {\tt missingPrerequisite(concept, level)}, where {\tt level} can be {\bf
    K}(nowledge), {\bf C}(omprehension), or {\bf A}(pplication).  {\tt
    missingPrerequisite} is inferred only in case of an insufficient result for
  an exercise related to the concept which is in focus.  This fact means that
  {\tt concept} is a prerequisite concept or item of the insufficiently 
mastered
  focus-concept and its {\tt level}-mastery in the user model is also
  insufficient. (The mastery of the focus-concept itself is not intermediate 
but
  immediate.)
  
\item {\tt SeenAndKnown(level)}, {\tt notSeenButKnown(level)}, {\tt 
SeenButUnknown(level)},
    {\tt notSeenAndUnknown(level)} are all stated for the focus-concept, where 
{\tt
    level} is one of the values K, C, or A.  {\tt xxUnknown(K)} means that in
  the user model the
  K-value (for the focus-concept) is insufficient (say, less than 80\%).
  {\tt xxUnknown(C)} means that in the user model the C-value (for the
  focus-concept) is insufficient but not the K-value.  {\tt xxUnknown(A)} means
  that in the user model the A-value (for the focus-concept) is insufficient 
but
  not the K- and C-values.

 The intermediate features are inferred for the {\tt focus-concept}, using the
 {\tt Seen} diagnoses of those items that contribute to the {\tt focus-concept}
 in the learning material and the {\tt K,C,A-mastery} for the {\tt
   focus-concept}.
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%
\section{Suggestion Mechanisms}
%%%%%%%%%%%%%%%%%%%%%%%%
A suggestor evaluates rules and writes its results on the SBB. Based on the
facts on the SBB, the actual suggestion presentation is generated to deliver

\begin{itemize}
\item a smile/noSmile puzzled, neutral, happy (smile), or sad (noSmile) face
  (which is thought as a shortcut). The default expression is neutral. \ednote{
    neutral wird als Standard angezeigt, puzzled by seltsamer navigationbesser
    waere das fuer unclear.}
\item the more detailed and personalized feedback that consists of a verbal
  feedback (only abbreviated in the tables below) and
\item   one or several presentation-actions.
\end{itemize}

We specified the reassuring 'smile' feedback as part of the GUI presented in
Figure \ref{fig:gi} because we feel that reassurance and positive feedback is
important for motivational reasons \cite{MandlGruberRenkl97} and for avoiding
situations in which the learner feels insecure.

  \begin{figure}
  \begin{center}
 \strut\epsfxsize=5cm\epsffile{gi.eps}
  \caption{The prototypical user interface for the global suggestions (TODO: 
 image with a suggestion.)}
  \label{fig:gi}
  \end{center}
  \end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Suggestors}\label{sec:suggestors}
%%%%%%%%%%%%%%%%%%%%%%%% 
The presentation-actions include 
\begin{enumerate}
\item 
navigation help 
\item content suggestions 
   \begin{itemize}
   \item present new or skipped example %(choice of difficulty)
   \item present similar example %later??: 
%   \item present counter example %later?? : \ednote{EA: Geht noch nicht}
   \item present new exercise %(choice of difficulty)
   \item present same exercise
%    \item  present comprehension question %later??:
%    \item present fill-in exercises %later??: 
    \item 
   present again the focus-concept maybe also examples, exercises 
   \item present (missed) instructional items\footnote{if
    misunderstanding of concept not attributed to prerequisites or reading 
time}
   \item  %WhichPrerequisites?  
     present certain prerequisites maybe together with examples and exercises 
   \item % Die K-C-A Present...
     K-present a concept. This means a presentation of items that intends the
     improvement of the user's knowlege of a concept.
     Mainly definitions and elaborations will be shown.
   \item C-present a concept. The goal of this action is to lead the user to an
     understanding of the concept. Currently, this is tried by presenting items
     like examples and understanding questions. \ednote{Eric: kannst Du die
       Verstaendnisfragen da auch noch hineinprogrammieren und vielleicht auch
       ein Beispiel bauen? EA: das muessen wir besprechen.}
   \item A-present a concept, i.e., present  some
     exercises and other items that help the user to learn how to apply the concept. 
   \end{itemize}
%\item  motivational support \cite{DelSoldato-its92,andereDieSoldatoNennt}
\end{enumerate}

For each learning-goal level (K-, C-, or A), a suggestion strategy can be
designed as a set of suggestors.  In what follows, we present the suggestors of
an {\em A-level} oriented suggestion strategy which (1) reacts to navigation
problems and (2) suggests content.  First, we explain the essence of each rule and then the
actual rules follow.
 
Note that $x$ in the expression {\tt seenButUnknown}($x$) denotes the missing
level of mastery of the focus-concept.


\paragraph{Rules for Navigation Suggestions} are needed because
{\activemath} delivers a hypertext learning document and it is known that
navigation in hypertexts needs special attention \cite{MurrayEtal00} and being
lost in hyperspace puts an additional load on the learner.

\begin{itemize}
\item if the user navigated appropriately, then provide reassuring
  feedback (abbreviated by 'smile').

\begin{verbatim}
IF Navig(okay) THEN - smile -  
\end{verbatim}
  
\item if an irrational navigation is diagnosed that started at point {\tt ?start} of the
  table of contents (TOC), then two pointers show the current
  position in the TOC and the {\tt ?start} position. In this case, the user can
  click the {\tt ?start} position to return to a 'useful' learning path.
  \ednote{EA->Erica: Das mit den Pointers gibts erst wenn die Messaging-Sachen
    fertig sind. Nicht für Koblenz. EM->EA: na, lassen wir es trotzdem. der report soll ja erst in
    2003 wirklich gelesen werden. Eric: aber wie zeigst du es jetzt an? Wird
    durch eine Seitenzahl angegeben. Muss noch debuggt werden.}

\small{
\begin{verbatim}
IF Navig(irrational,?start) THEN - noSmile -
                                 ``did you get lost by chance?''
                                 pointer(current) and pointer(?start)
\end{verbatim}
}
\end{itemize}
\ednote{soll smile bleiben, oder happy? muss curretn gross geschrieben werden,
  weil es ein parameter ist oder als Variable ?current ??}

\paragraph{Rules for Content Suggestions} are needed because if the
goal-level of mastery is not yet reached by the learner, then the presentation
of appropriate content might help to improve.  As opposed to the local feedback
that corrects single problem solving steps, the global feedback described below
prompts and supports the learner in activities such as reading, repeating,
self-explaining, comparing, varying, information gathering that are known to
improve learning, see, e.g.,
\cite{ChiDeLeeuwChiuLavancher94,AlevenKoedingerCross99,Renkl01,Stark00b,ChiDialog01}.
   
\begin{itemize}
\item If the last exercise was solved correctly and the concept in focus is
  mastered, then provide reassurance. 
\begin{verbatim}
IF Known(A ?focus)                        THEN - smile -
\end{verbatim}
  
  A second rule belongs to a more elaborate strategy that assumes that even if
  the mastery-level is reached, more exercising can strengthen the mastery of
  the concept and the confidence for less confident learners.
This rule requires that, if there is an exercise that is more
  difficult than those solved already, then present it.

  \ednote{Merken:  wenn mastery-level
    erreicht, musste man doch nicht noch weiter machen. Das System koennte dann
    bestenfalls den button 'give me more' mit der Regel unten vorsehen???..Ich
    versuche mal eine Erklaerung. Es waere auch schoen, wenn der give-me-more
    button adaptive waere und in so einem Fall eine schwierigere Uebung
    heraussuchen wuerde. Jedenfalls sollen nicht unendlich viele Vertiefungen stattfinden.} 

\begin{verbatim}
IF Known(A ?focus)                        THEN - smile -
AND solution(?id correct)                      ``see more''
AND exerciseFor(?focus ?id)                    present(?exc1)
AND exerciseFor(?focus moreDiff(?id ?exc1))
AND notSeen(?exc1)
\end{verbatim}
  
\item If seenButUnknown(A) holds for the focus-concept, then present examples
  similar to the failed exercise of the focus-concept, unseen simpler 
exercises,
  and then again the incorrectly solved exercise of the focus-concept.
\ednote{Eric: Merken (Diagnose noch zu unspezifisch): die Regel ist
  debatierbar...denn der erste und zweite Vorschlag 
  koennten aus 2 verschiedenen Regeln resultieren. ZB. koennte man es zunaechst
  mit einem aehnlichen Bsp versuchen und wenn das nicht hilft, noch mal mit
  einfacherer Loesung oder umgekehrt, wenn man das mit der Motivation ernst
  nimmt. Die Reihenfolge koennte auch davon abhaengen, wie sehr falsch die
  Loesung von id war. wenn nur wenig falsch, dann kann der Vergleich mit einem
  aehnlichen Bsp schon helfen, sonst eher nicht. Haengt auch noch davon ab, ob
  so ein Beispiel bzw Uebung in der DB ist.  ist jetzt so
implementiert? }
%
This suggestion is made because A-mastery is the learning goal but not yet
achieved, and therefore another exercise for the focus-concept should be offered
to be solved. This exercise should be a bit simpler in order to keep the user's
motivation up (in the proximal zone of development \cite{Vygotsky78}). Then an
example similar to the exercise should be shown for comparison. Finally, the
originally failed exercises should be presented again.



\small{
\begin{verbatim}
IF SeenButUnknown(A ?focus)       - noSmile -
AND solution(?id incorrect) THEN  ``go deeper into the problem''
AND exerciseFor(?focus ?id)       exerciseFor(?focus lessDiff(?id ?exc1))
                                  exampleFor(?focus simTo(?id ?exm1)) 
                                  present(?exm1 ?exc1 ?id)
\end{verbatim}
}
\item If seenButUnknown(C) holds for the focus-concept, then show not yet
  sufficiently seen examples and counter-examples (if available) and ask for
  an explanation of the examples.  
  
  This suggestion is made because C-mastery of the focus-concept is not yet
  achieved and therefore, this comprehension has to be supported. This is tried
  by showing more examples and prompting the learner to engage herself in
  self-explanation.

\begin{verbatim}
IF SeenButUnknown(C ?focus)        - noSmile -
AND exampleFor (?focus ?exm) THEN  ``please explain example''
AND NOT Seen(okay ?exm)            present(?exm)
\end{verbatim}

\comment{
\item If seenButUnknown(C) holds for the focus-concept and
 an error/counter-property (cp) is diagnosed in the solution of the exercise 
(id) for which a
counter-example can be given, then present an example and a counter-example for
the focus-concept and ask the learner to compare them.

This suggestion is made because the C-mastery  of the
focus-concept is not yet achieved and therefore, this comprehension has to be
supported. Since a misconception\ednote{Misconceptions haben wir ja 
noch
  nicht...muss man sich nochmal ueberlegen.} (cp) is diagnosed for which a 
counter-example is
available, the support is offered by showing an example and a counter-example 
in
parallel and ask the learner for a comparison that should yield deeper
comprehension and learning.

\begin{verbatim}
IF SeenButUnknown(C ?focus)       - noSmile -
AND solution(?id ?cp)       THEN ``please compare ?id with ?exm and ?ce''
AND exerciseFor(?focus ?id)       exampleFor(?focus simTo(?id ?exm))
                                  counterExampleFor(?cp,?ce)
                                  present(?exm + ?ce)
\end{verbatim}
}
\item If seenButUnknown(C) holds for the focus-concept then prompt the learner
  to explain and vary an example of the focus-concept.
This suggestion is
  made because C-mastery  of the focus-concept is not yet achieved and 
therefore,
  this comprehension has to be supported. This is tried by
  showing more examples and prompting the learner to engage
  herself in  self-explanation.

\begin{verbatim}
IF SeenButUnknown(C ?focus)        - noSmile -
AND exampleFor (?focus ?exm) THEN  ``please explain and vary example''
AND  Seen(okay ?exm)               present(?exm)
\end{verbatim}
  
\item if seenButUnknown(K) holds for the focus-concept, then the user should
  re-read the last page on which the concept has been elaborated.
  This suggestion
  is made because K-mastery (and also A- and C-mastery) is not yet achieved.
  Since A-mastery is the learning goal in the overall strategy, everything
  (reading, examples, and exercises) for the focus-concept, except any solved
  exercises, needs to be repeated. 

\begin{verbatim}
IF seenButUnknown(K ?focus) THEN - noSmile -
                                 ``please return to last page''
                                  present(lastPage)
\end{verbatim}
% and 
% \begin{verbatim}
% IF seenButUnknown(K ?focus)
% AND solution(?id correct)   THEN  REJECT present(?id)
% \end{verbatim}

\item if notSeenAndUnknown(K/C/A) holds for the focus-concept, then
  K/C/A-present, respectively.  This suggestion is made because A-mastery is 
the
  learning goal and so content for every level not mastered yet is suggested
  again.

\begin{verbatim}
IF notSeenAndUnknown(K/C/A) THEN ``repeat''
                                  K/C/A-present(focus)
\end{verbatim}

\item if there is a missing K/C/A-prerequisite $c$ (that is, a prerequisite
  concept that whose mastery level has not reached K/C/A), then
  K/C/A-present that prerequisite $c$ according to what is missing for the 
mastery.
No further substantiation needed because the failure is likely to be caused by
the missing mastery of the prerequisites of the focus-concept.

\begin{verbatim}
IF missingPre(?c, ?level)   THEN -noSmile -
                                 ``missing prerequisite''
                                 ?level-present(?c)
\end{verbatim}
\item if an item ID has been seen sufficiently and more than twice, then
  do not present automatically  ID again. Otherwise the motivation might drop.
This selection is made in the suggestion generator.
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Conflict Management}
%%%%%%%%%%%%%%%%%%%%%%%%
%wieso
It is possible that conflicting suggestions occur on the
SBB, in case a user action triggers several rules. For instance, when a new 
page is selected by the user, this can trigger a navigation suggestion as well as
a conflicting concept presentation suggestion. \ednote{Der Konflikt liegt 
darin, dass nicht alle Vorschl\"age gleichzeitig angezeigt werden k\"onnen. Richtig 
konfligierende Diagnosen auf dem DBB habe ich noch nicht entdeckt, ich finde auch kein
  Beispiel daf\"ur. EM-> will man diese dann nicht gleichzeitig anzeigen - 
frage bitte  Markus von ergosign.}
%
If not all of the suggestions can be presented at the same time, then a rating
has to indicate the priorities, and only the rules with the highest rating 
will generate their
suggestions.

For the decision it makes a difference whether two suggestions are both of the 
same class,
say navigation suggestion, or not.  Therefore, the rules are classified as {\em
  navigation}, {\em content}, and {\em exercises} rules, currently. A conflict
between two rules from the same class is resolved by a default resolution 
(e.g.,
more specific rules have higher rating). A conflict between rules from 
different
classes is resolved based on a particular conflict resolution strategy. The
simplest strategy decides according to a fixed priority rating of the classes.
If the highest rated rule cannot generate a suggestion, e.g., because
there is no counter example, then the it gives up and the next best rated rule 
takes the turn.

For the implementation of a conflict resolution strategy in {\activemath} the
{\sc Jess} \cite{Jess}-interface {\tt strategy} and its method {\tt compare} are 
employed. The
{\tt strategy} can be easily configured in the {\activemath} property list.

% %Beispiele aber lieber doch nicht
% Some suggestions can be given concurrently.
% For instance, a navigation correction and missing prerequisites presentation may
% be suggested concurrently, simple and elaborate feedback can be suggested at the
% same time and the same holds for immediate and delayed feedback.  
% %Beispiel
%  A conflict resolution is needed, if the
% SBB contains the feedback suggestions smile and noSmile at the same time.
% The  suggestion to read about a concept versus a suggestion to offer
% an exercise have to be rated. 

%%%%%%%%%%%%%%%%%%%%%%%%
\section{Implementation}\label{sec:implementation}
%%%%%%%%%%%%%%%%%%%%%%%%
The described mechanisms are implemented by a {\sc Jess}-application. 
This section covers the interfaces to other components of {\activemath} as 
well as internal reasoning mechanisms. The suggestion mechanism runs in three phases.
First, the evaluators communicate their results to a diagnosis
component. This potentially triggers the inference of intermediate 
diagnoses/facts
(1. phase).
%
Then, suggestions are generated from the intermediate diagnoses and put on the
SBB (2. phase).
%
The third phase resolves existing conflicts and realizes the presentation of
 content if needed. 

The next subsections go into more detail.

%%%%%%%%%%%%%%%%%%%%%
\subsection{Interfaces}
%%%%%%%%%%%%%%%%%%%%
\label{sec:interfaces}
We need to consider several interfaces. First we have to look at the evaluators,
because their diagnosis blackboard generates the diagnoses needed by {\nextbest}
to make the suggestions. Then, we will have a closer look at the {\nextbest}'s
interface and the functionality it offers. Finally, we'll briefly discuss the
presentation servlet.

\subsubsection{Evaluator Interface}

As the evaluators have to be notified about the user's actions their interface
is based on methods designed to do this job. There are some central actions a
user can perform in {\activemath} currently. Each is reported using a corresponding
method:

\begin{description}
\item[{\tt startPageBrowsed}] the user has come to a new page.
\item[{\tt endPageBrowsed}] the user has finished  a page.
\item[{\tt startExercise}] the user started an exercise.
\item[{\tt endExercise}] the user finished an exercise.
\item[{\tt enableEyeTracker}] the user is using the poor man's eyetracker
\item[{\tt elementHasBeenSeen}] the user has seen an item on the current page 
(this is
  used only when the poor man's eyetracker is enabled).
\end{description}

This interface provides access the different evaluators, it will propagate the
method calls to those evaluators  registered for each action. 
For instance, the information in the {\tt endExercise} method is only relevant
to the exercise evaluator, so the other evaluators need not to be notified of
this.
In contrast to the {\tt endExercise} method the output of {\tt endPageBrowsed}
is needed both, by the read and the navigation evaluator.

For a more technical description of these methods,  see\\
{\tt http://www.activemath.org/\~{}ilo/java-doc/org/activemath/abstractcontent/\\
AbstractUserModel.htm}.


\subsubsection{The Suggestion Mechanism Interface}
The suggestion mechanism has to be notified, when an evaluator delivers a
diagnose.  This is realized by the {\tt addDiagnose} method, which will insert
the diagnose into the suggestion blackboard. When the mechanism generates a
suggestion, it notifies the servlet that deals with displaying and rendering.
This servlet receives the suggestion that is to be presented by the method {\tt
  getNextBestAdvice}.

Since the user can also actively request some advice, the servlet needs a
possibility to tell {\nextbest} to generate a suggestion. {\tt
  requestNextBestAdvice} is the method to be called in this case.  Summarizing, we have
the following methods in the interface
\begin{description}
\item[{\tt addDiagnose}] to inform {\nextbest} about new diagnoses
\item[{\tt getNextBestAdvice}] to allow the presentation servlet to access the 
suggestion
\item[{\tt requestNextBestAdvice}] to explicitely request a suggestion.
\end{description}

\subsubsection{Presentation Interface}
The presentation servlet is accessible for other modules via {\tt
  HTTP}. It realizes the following functionalities
\begin{itemize}
\item visual notification that a new suggestion is available
\item display of the suggestion text
\item display of the suggestion content
\item pass an active user request for  the generation of a suggestion.
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{The Diagnosis Blackboard}
\label{sec:dbb}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
There is one DBB per user. This allows to keep the {\sc Jess} fact base clean and to
speed up the suggestion generation.  A DBB receives input from the evaluators,
reasons about this input, and then transmits its diagnoses to the SBB as
described in the following.

Once the evaluators have processed the raw input coming from the user-tracking
servlets, they pass first diagnoses to the DBB.\ednote{ Die evaluatoren bekommen
  ``blanken `` Input vom User-tracker. Diese werden dann aufbereitet, einfache
  Sachen schon in dem Evaluator selbst berechnet, dann wird zum Schluss ein fact
  in Jess gelegt} For instance, if the student finished an exercise, the {\tt
  ExerciseEvaluator} is notified by the user-tracking device with a call to {\tt
  endExercise}, it performs some simple evaluations, and then inserts a new fact
into the DBB, namely
\begin{verbatim}
(solution <exercise> <rating>),   
\end{verbatim}
where {\tt <exercise>} is the exercise's ID and {\tt <rating>} is one of the
values {\tt POOR}, {\tt OK} or {\tt GOOD}. 

The DBB contains a {\sc Jess} rule engine to reason about the incoming facts.
Its rules infer the intermediate facts in the DBB.  For instance, if the
performance was rated {\tt POOR}, the rule engine will try to find out why the
student performed poorly. A reason could be that one of the prerequisites of
this exercise was unknown to the student. In this case, a rule will fire and
insert a new fact into the DBB fact base. The concrete fact is
\begin {verbatim}
(missing-prerequisite <dimension> <exercise> <item>),
\end{verbatim} 
where {\tt <dimension>} is the target mastery level, {\tt <exercise>} is the ID
of the exercise for which the missing prerequisite was found, and {\tt <item>}
is the ID of the missing prerequisite. All the currently available DBB-facts can
be found in table~\ref{fig:facts}.
 
\begin{table}
\begin{tabular}{|l|l|p{6cm}|}
\hline 
Fact & Options & Interpretation \\
\hline 
\hline 
PRIMITIVE FACTS & &\\
\hline

%% NAVIGATION
navigation & {\tt OKAY} & student is aware of her position in the course 
\\ 
 & {\tt IRRATIONAL}, {\tt start} & student navigates
 without following any dependencies or following the outline of the course 
since position {\tt start}\\ \ednote{Merken: erst bei Multimodal woher bekommt das nextBest dann
  den zweiten pointer - im Augenblick gar nicht - wo der user jetzt ist? was ja
  in den Regeln vorkommt}
\hline

%% ELEMENT HAS BEEN SEEN
seen & {\tt item}, {\tt NOTSEEN} & {\tt item} was on the last page,
but the student did not read it at all.\\
 & {\tt item}, {\tt SHORT} &  {\tt item} was on the last page, and the
 student did not read it long enough.\\
 & {\tt item}, {\tt OK} &  {\tt item} was on the last page, and the student
 has read it long enough.\\
 & {\tt item}, {\tt LONG} &  {\tt item} was on the last page, and the
 student has read it far longer than the average reading time.\\
\hline

%% EXERCISE SOLUTION
solution & {\tt exercise}, {\tt POOR} & the student performed poorly
in an exercise {\tt exercise}.\\
 & {\tt exercise}, {\tt OK} & the student performed neither well nor poorly in an
exercise {\tt exercise}.\\
 & {\tt exercise}, {\tt GOOD} & the student performed well in an exercise {\tt exercise}.\\
\hline
\hline
INTERMEDIATE FACTS & &\\
\hline

%% MISSING PREREQUISITES
missing-prerequisite & {\tt K}, {\tt exercise}, {\tt item} & {\tt item}
is a missing prerequisite for the student to solve {\tt exercise}
because she does not know it well enough.\\
 & {\tt C}, {\tt exercise}, {\tt item} & {\tt item}
is a missing prerequisite for the student to solve {\tt exercise}
because she does not truly understand it.\\
 & {\tt A}, {\tt exercise}, {\tt item} & {\tt item}
is a missing prerequisite for the student to solve {\tt exercise}
because she cannot apply it properly.\\
\hline
%% SEEN BUT UNKNOWN
seen-but-unknown & {\tt K}. {\tt concept} & {\tt concept} has been seen several times but
the user still does not know it.\\
& {\tt C}. {\tt concept} & {\tt concept} has been seen several times but the 
user still
does not understand it.\\
& {\tt A}. {\tt concept} & {\tt concept} has been seen several times but the 
user still
cannot apply it.\\
not-seen-and-unknown &{\tt item} & The user has finished with a page but its
focus-concept hasn't been seen and is not even known.\\ 
\hline
\end{tabular}
\caption{The DBB facts with their parameters}\label{fig:facts}
\end{table}

\paragraph{Technical Description}
The Java code in the DBB serves as a wrapper for the rule engine {\sc Jess}, a
Java Expert System Shell.  In the following, we present
selected rules that run on the DBB, namely those needed to infer that the
student missed a prerequisite to solve an exercise.

To continue the example, the search for missing prerequisites starts, when the 
fact {\tt (solution exercise POOR)} is added to the fact base.\\
This triggers a rule that queries the user model for the student's mastery 
values of the prerequisites of the exercise.  Both, the concepts that the
exercise is {\tt for} as well as the concepts the exercise
{\tt depends-on}, are needed in the general diagnosis procedure. In our example, 
we only need the mastery values of concepts that the exercise depends on (these are 
precisely the
prerequisites) to diagnose prerequisites that are insufficiently known. 
% Das Idee ist folgende. <depends-on> 
%gibt ja diejenigen
%  Konzepte, von denen die Aufgabe abhaengt, also die Vorbedingungen, waehrend 
%<for> das
%  Konzept gibt, wofuer die Aufgabe gedacht ist. In dem Beispiel, das wir hier 
%durchgehen,
%  wollen wir gerade fehlendes Vorwissen entdecken, brauchen also nur 
%<depends-on>. Mit
%  Hilfe von <for> kann man z.B. herausfinden, dass der Benutzer eine Aufgabe 
%schlecht
%  geloest hat, obwohl er das Ziel der Aufgabe schon sehr gut kennt. EM-> Eric:
%  ja schreibe es bitte auf. Ich verstehe immer noch nicth, wieso beide 
%Funktionen in
%  derselben Regel stehen.}

\newpage The {\sc Jess} rule that collects the concepts the exercise depends on
is given in the following.\footnote{The syntax is simplified. ?xx indicates that
  ?xx is a variable, ;; declares comments.}
  
\begin{verbatim}
(defrule start-performance-diagnosis
 ;; IF
 ;; Student has solved exercise ?exc with success rating ?r
 (solution ?exc ?r)

 ;; AND
 ;; The reference to the user model is ?model
 (model ?model)              
  
=>

 ;; THEN
 ;; Load the mastery-values of ?user for exercise ?exc was for
 ;; from ?model into the the fact base
 (get-for-what ?exc ?user ?model)  

 ;; AND
 ;; Load the mastery-values of ?user for prerequisites of exercise ?exc
 ;; from ?model into the the fact base
 (get-prerequisites ?exc ?user ?model) 
)
\end{verbatim}
 The rule calls  the
functions {\tt get-prerequisites} and {\tt get-for-what} which load the
student's mastery values of  concepts into the {\sc Jess} fact base.

Specifically, {\tt get-prerequisites} queries the knowledge base {\mbase} for
the concepts {\tt exercise} depends on, that is, its prerequisites, 
and inserts facts such as {\tt
  (dependency ON MONOID-EXERCISE DEF-MONOID)} which expresses that the {\tt
  MONOID-EXERCISE} needs the concept defined in {\tt DEF-MONOID} as a
prerequisite.
{\tt get-for-what} fetches the concepts {\tt exercise} is for. 
We say that those concepts are the {\em goals} of the exercise.  

Then, the DBB queries the user model for the user's mastery values and new 
facts such as {\tt (mastery-value K DEF-MONOID 0.4)} are asserted.  This fact states
that the likelyhood that the learner {\tt K}nows the concept Monoid is 0.4.
Once the fact base contains the needed information about the student's mastery
of the related concepts, the next rule that fires is {\tt
  poor-perf-poor-application} that inserts new intermediate diagnoses into the
DBB.

\begin{verbatim}
(defrule poor-perf-poor-application
 ;; IF
 ;; The student has given a POOR solution to exercise ?exc
 (solution ?exc POOR)

 ;; AND
 ;; exercise ?exc depends on concept ?id
 (dependency ON ?id ?exc)   
 ;; AND 
 ;; The application value for item ?id is low (below 0.3)
  (mastery-value A ?id ?val &: (< ?val 0.3))

=>
  
 ;; THEN
 ;; assert that the missing prerequisite item ?id  has been found for ?exc.
 ;; There was a lack at the application-level. 
 (assert ("missingPrerequisite" "A" ?id ?exc))
) 
\end{verbatim}
Now those diagnoses can be processed by the knowledge sources that insert facts
into the suggestion blackboard. The following rule performs such a transition to the SBB.

\begin{verbatim}
(defrule diagnosed-missing-prerequisite

  ;;NotSeenAndUnkown holds for the focus concept
  ;; Our missing prerequisite
  ("missingPrerequisite" "A" ?id ?exc)
  ;; The username
  (user ?user-name)
  ;; Reference to the suggestion blackboard.
  (nextbest ?sbb)

=>

  ;;Send a suggestion to a-present the ?id for ?user because it is a missing precondition  
  (suggest ?user ?sbb ("A-present" ?id "You should have a look at some items that are 
needed to solve this exercise which you are not comfortable enough with" "MISSING_PRECONDITION"))
)
\end{verbatim}

%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{The Suggestion Blackboard}
\label{sec:sbb}
%%%%%%%%%%%%%%%%%%%%%%
The suggestion rules watch the diagnoses on the DBB and put new facts on
the SBB. The rules compute the suggestions and their presentation via {\activemath}
  with a specialized course generator. Moreover, the SBB can resolve
potential conflicts between suggestions.
% This is done by using the {\tt NextBestPP}, a subclass of
%{\activemath}'s course generator.

Let us continue the example from the last section. Once the fact 
\begin{center}
{\tt ("A-present" ?id "You should have a look at some items that are 
needed to solve this exercise which you are not comfortable enough with" "MISSING-PRECONDITION")} 
\end{center} 
has been put onto the SBB, 
% has been put onto the SBB, das nur, weil diagnosen auf SBB kopiert
% werden. Denn die Regeln können in Jess nicht ZWISCHEN fact basen arbeiten.
% EA: ACHTUNG! Ich habe den Mechanismus umgebaut: Es gibt solche transitionsregeln, die etwas vom DBB benutzen, um etwas aufs SBB zu schreiben!
%   der o.g. Umbau. Hat auswirkungen auf Konfliktresolution:
% Mehr Typen von Regeln (insbesondere ist missing-precondition einer.)
%
%  verschiedene Fehler koennten doch auf missing-prerequisite deuten und dann
%  selbe reaktion erzeugen?}
a suggestion rule fires that calls presentation functions for content
generation:
% wieso bezieht sich die naechste Regel nur auf EXERCISES?
\begin{verbatim}
(defrule do-a-present
 ;;IF
 ;;there is a diagnose yielding the need to a-present ?id
 ("A-Present" ?id ?message ?type)

 ;;AND
 ;;The Nextbest object is ?nextbest. This is needed to pass the resulting 
 ;; suggestion. 
 (nextbest ?nextbest) 

 ;;AND 
 ;;the username is ?user
 (user-name ?user)

=>

 ;;THEN

 ;;set the correct presentation scenario 
 (bind ?scenario "A-present")

 ;;create a nextbest presentation generator for the item ?what,using the 
scenario ?scenario
 (bind ?nextbest-pp (new NextBestPP ?id ?user ?scenario ))   

 ;;run it
 (call ?nextbest-pp run)

 ;;get the generated document
 (bind ?document (call ?nextbestb-pp getStaticBook))

 ;;generate the Suggestion Object
 (bind ?suggestion (call ?nextbest createSuggestion))

 ;;set the expression of the little face
 (call ?suggestion setMood "unhappy")

 ;;set the message that should be displayed
 (call ?suggestion setMessage ?message)
  
 ;;set the ?document as content
 (call ?suggestion setPresentation ?document)

 ;;set the type of the suggestion to be ?type
 (call ?suggestion setType ?type))

 ;;tell nextbest to send the suggestion to the presentation servlet.
 (call ?nextbest makeSuggestion ?suggestion)
)

\end{verbatim}
When the generation process is finished, {\nextbest}'s graphical user interface
displays a text saying \(<\)"You should have a look at some items that are
needed to solve this exercise which you are not comfortable enough with"\(>\)
followed by a generated {\html} link to the suggestion's content.

\paragraph{Technical Description}
\label{sec:sbbtp}

The SBB is implemented using a {\sc Jess} engine wrapped by 
Java code that controls the access.
The interface 
of the SBB is the following.

\begin{verbatim}
interface AbstractNextBestCentral 
{  
  getNextBestAdvice(id, user);
  
  createNextBestFor(user);
  
  requestNextBestAdvice(user, topic);
  
  addDiagnose(user, factname, args);

  public void suggestionFollowed(String user, int suggestion_type);

  public void suggestionIgnored(String user, int suggestion_type);
}
\end{verbatim}

%%%%%%%%%%%%%%%%%%%%%%%%
\section{Design of the Suggestion User Interface}\label{sec:design}
%%%%%%%%%%%%%%%%%%%%%%%%bis hier
The primary user interface of the global suggestion mechanism consists of a 
face/companion with a
variety of possible types of states. This face moves regularly even if the
user does well and there is no new suggestion. This is implemented in order to prevent the
user wondering about the mechanism being stuck.
%
This face informs the user whether the system wants to offer a suggestion or
not. If not, it reassures the learner by looking happily.

When new coaching can be suggested by the system, the face changes and the
student can now choose to follow or to ignore the suggestion. The offer 
 expires after some time and then disappears.  
%
The suggestions are personalized in that the messages address the
learner by the first name she has introduced at login.

To follow the offer, the learner presses a button for requesting the detailed
suggestions.  Then, a new browser window pops up that contains the generated
content.

% \ednote{Merken:
%   implementiert haben!!! 1. verschiedene, sich regelmeassig bewegende 'Gesichter'
%   2. TOC mit pointers? nicht Koblenz, 3. active-request buttons?
%   Antwort: 1. Ist auf jeden Fall machbar, vielleicht in Koop. mit
%   Manolis/Duncan. 2. Sofort und einfach realisierbar sobald MiDis Messaging Architektur implementiert
%   ist. 3. 2 Stueck gibt es schon. Ergosign schlägt ruhig mehrere vor}

The user interface also includes buttons for requesting more specific
information and content as described in \ref{sec:passiveactive}.

% \ednote{falls aktives nie (oder 5 mal nicht?) benutzt wird, dann nur noch
%   passive und Wiederbelebung auf explizite ANfrage? ergosign:
%   Keine Veränderungen!!}
%%%%%%%%%%%%%%%%%%%%%%%%
\paragraph{Active vs.~Passive Feedback}\label{sec:passiveactive}
%%%%%%%%%%%%%%%%%%%%%%%%
Several empirical investigations \cite{ButlerWinne95,ReiserEtal94} found that
feedback on demand (we shall call it {\em active} feedback here) typically
yields longer learning time than automatic feedback. However, feedback on demand
yields deeper knowledge, i.e., better transfer ability and more self-regulation,
-monitoring, and motivation.

Therefore, the global feedback GUI in {\activemath} has two different components: general 
feedback is provided that the learner receives rather passively as well as
buttons for more specialized requests and for 
more specific feedback on demand.  {\activemath} realizes the active feedback 
by buttons that
the user can click on, whereas passive feedback means the user does not request
the information but receives it automatically in the NextBest window through an
initiative of the system.  

777user chooses time and type of information and therefore needs to think about
her learning requirements more actively in case she requests more specific
information. Not invasisve in both cases

%
 In the passive mode, if the user does not click a specific button (i.e., she is
 passive), there are two alternatives. Either the system presents the highest
 rated amongst all suggestions to the user or a dialog is started to find out
 what would be the appropriate and desired help. The latter way will be explored
 in the future only.

% one BUTTON corresponding to each one suggestion mechanism? 
 The first button for a specific global feedback request implemented in
 {\activemath} is the {\sc giveMeMore} button. The learner can use it to ask for
 more exercises, similar to one she worked on recently, or for a similar
 example.

We plan to design other buttons with a telling name for each of the suggestion
mechanisms \ednote{Eric: hat ergosign da etwas gesagt? frage bitte noch mal
  edxplizit weil es sonst wohl vergessen wird}, e.g., whereAmI?, 
DoIKnow?,
CorrectSolution? LookUpYourUserModel!,
WhichPrerequisites?, explainMore!  This is a first step towards a tutorial
dialog.

%%%%%%%%%%%%%%%%%%%%%%%%
\section{Future Work}\label{sec:future}
%%%%%%%%%%%%%%%%%%%%%%%%
Some future work has been mentioned already. In this section, we describe the
plans more systematically. In particular, we mention the extensions and
improvements of the diagnosis and suggestion mechanisms including the
consideration of the learner's motivation and different suggestion strategies
for different learning goal-levels and for different learning scenarios.
Moreover, we discuss the empirical investigations that will be conducted in
cooperation with psychologists.

\subsection{More Elaborate Diagnoses}
The duration of reading is one of the data from which diagnoses can be inferred.
However, an individual learner may have an individual average reading speed.
Therefore, the {\em optimal reading} time has to be determined for each learner
during the time the student uses {\activemath} in order for the reading
evaluations to become more precise.

%\paragraph{Omega CAS Diagnoses}
We plan to develop the more specific diagnostic features of a proof attempt with
the proof planner Omega and with Computer Algebra systems.
\begin{itemize}
\item More elaborate {\tt A-mastery} values for all
  the concepts involved in solving an exercise  might  be determined from the 
user's detailed
  proof attempt and from the difficulty of the exercise. This requires to model
  the problem solving process, e.g., by a  dynamic Bayesian Net  which 
contains the
  concepts and
  methods  involved in  the exercise as nodes and its various results as 
symptom nodes.
 
\item From such a dynamic
  Bayesian Net  the missing prerequisites may be diagnosed more properly.
\item The ability-value of using Omega or a particular CAS will be updated
  (e.g., from the number of syntax errors). 
\item An update of the {\tt guess} and {\tt slip} values could be
  learned.
\item If the author provides a tuple (typicalError, teacherDiagnosis,
  teacherFeedback, teacherReaction), then an evaluator can pass the
  (teacherDiagnosis, teacherFeedback, teacherReaction) tuple to the diagnosis 
blackboard in case
  the user's input matches typicalError. \ednote{Eric: das gibt es bei 
Barbara
    eingeschraenkt und lokal. Wollen wir das auch global? Insbesondere ist
    teacherFeedback, teacherReaction global oder nur lokal}
\end{itemize}

%evaluators for MCQs
\ednote{GG: was gibt es davon schon?999}
With more effort for authors, a more detailed diagnosis of the user's solution
for multiple choice questions could be provided in the future. For instance,
%
(1) if the author of the alternative answers in MCQs follows a particular
Boolean schema with mastery variables for several concepts, then this schema 
can
be used for an automatic diagnosis that computes a mastery diagnosis for 
several
involved concepts rather than for one concept only. Such a schema consists of
Boolean combinations of the mastery\footnote{k-,c-,or a-mastery} of
the involved concepts for each answer.
%
(2) \ednote{GG fragen} If the author provided the information {\tt
  (teacherDiagnosis,  teacherAction)} for each answer of an MCQ,
then the proxy can pass this information to the diagnosis blackboard.
\ednote{muesste realisiert werden falls sinnvoll Adrian} 

\ednote{999 GG:  Woher soll das in MCQs
  kommen? auch ein eval code? oder nur teacherDiagnosis? oder soll da
  nur zugelassen sein, dass eine Diagnose so wie k-mastery=low als DiagnoseBB
  geschickt wird und die suggestors Vorschlaege daraus ableiten??  correct
  solutions gezeigt in  NextBest  weil Ich
  glaube, erst in NextBest, weil sonst keine ANstrengung des Benutzers noetig}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{More Suggestions} %more suggestion 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Our next investigations will focus on the design of a
personalized and  gender-specific verbalization of the feedback and on the
design of new validated rules as well as on other kinds of
global feedback including
\begin{itemize}
% \item 
% with a progress of {\activemath} we shall introduce new types of exercises.
% \ednote{Eric, Erica: was ist mit der AUfforderung CompareOwnSolutionWithExpert! ? Muss als
%   exercise praesentiert werden (mit evaluation?)
%   Auch  whereIsError? CompareYourAndExpertSolution!
%   ExplainExample! Summarize! Misconceptions? DrawConceptMap! auch mit Blooms
%   prompts. Hier sollen im globalen feedback vor allem solche praesentiert
%   werden, die einen delay brauchen (zB JOL wo eigene Einschaetzung involviert ist)}

\item exploration tasks with system support (dependencies, more information,
  e.g.,historic information) %ExplainMore!   
\item  meta-reasoning exercises, for instance \begin{itemize}
  \item structure a task or solution %(e.g., if no understanding of a task) 
\ednote{ask Grabowski 999}
  \item self-explaine
  \item exercise/example variation
  \item devise examples supported by the system
  \item compare solutions supported by the system
  \item draw concept map
  \end{itemize}
\item feedback for improving motivation. 
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Reactions to the Learner's Motivational 
State}\label{sec:motivation}
%%%%%%%%%%%%%%%%%%%%%%%%
%motivational principles
Few tutor systems take the learner's motivational state into consideration
currently.  For instance, WEST \cite{BrownBurton82} follows the principle {\em
  Do not tutor on two succcessive moves, no matter what} which tries to avoid
demotivation by too much interference. SOLA \cite{Arshad90} structures content
according to the learner's style (holistic vs. serial and low vs. high
confidence). {\em Less confident students receive content focussed on one major
  topic, whereas highly confident students may focus on several topics; a
  serialist will rather focus on one topic whereas a holist could pursue several
  topics at the same time}. For motivational reasons, MENO \cite{Woolf84}
delivers a particularly positive feedback, when a student eventually succeeds
who has previously failed in order to restore her confidence. This is -- as a
tendency -- particularly important for female students.

Those and similar principles such as {\em Do not present new material, when the
learner is engaged already}, {\em Once a less confident learner does a task
  well, activities that are likely to succeed should be presented by the
  system}, and {\em Highly motivated students deserve praise even if their
  performance is not optimal} \cite{soldato:aied89} have to be tested empirically
and will be formalized in {\activemath}' suggestion rules.

%techniques to enhance motivation
%was ist davon MOTIVATION ATTENTION EXPECTATION AFFECTION?? attention,
%curiosity, challenged, bored, frustrated phantasy, satisfied control??999
%
Several techniques to enhance the learner's motivation are known. For 
instance,
curiosity can be stimulated if necessary for increasing the learner's 
motivation.
%
Perceptual curiosity techniques deliver audio and visual efects such as
highlighting a detail, i.e., focussing the learner's attention to the
detail. Also bright colors and animations focus attention. 
%
Cognitive curiosity can be stimulated by puzzling questions, incomplete
information, relevant analogies, counter-examples, unexpected or paradoxical
questions, or a topic that requires an explanation.
%
In order to reduce the learner's frustration, forthcoming difficulties
  should be signalled.
%
An anti-help suggestion button should allow the student to control the 
provision
of suggestions.

\paragraph{Diagnosis of Motivational State}
%diagnosis of motivation
In order to be able to react to the learner's motivational state it has to be
diagnosed in the first place.  A reliable indication for the student's 
intrinsic
motivation is the student's effort rather than her performance
\cite{DelSoldato-its92}.
%
A motivation diagnosis can also be obtained by self-evaluation questions
combined with inferences. For instance, low confidence is inferred, if the 
student
repeatedly asks for help and high confidence is inferred in case of total
absence of help requests.
The absence of help requests may indicate that
  problem is too easy. Motivated students may use hints  or not even request 
help.
Students who ask for help without exploring the hints provided by the
  system are likely to be demotivated.

% To summarize which information about the user is needed to correctly diagnose
% her motivational state
% \begin{itemize}
% \item learner style: holistic vs. serialist
% \item confidence 
% \item frequency of help request
% \item acceptance/refusal of suggestions
% \item intrinsic effort
% \item ?control
% \item ?challenge
% \item ?curiosity
% \end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Different Pedagogical  Strategies}
%%%%%%%%%%%%%%%%%%%%%%%%
%strategie abhaengig
The computation, rating, and eventual choice of a suggestion depends on the
pedagogical strategy whose choice, in turn, may depend on the user's ability,
goals, scenario, and motivation.  For instance, a good learner will more likely
succeed with a few hints and active learning and benefit from it, whereas a 
weak
learner has to be corrected immediately and strongly guided.

A strategy (represented as a set of rules) determines, e.g., when, how often,
and which types of exercises and examples to present, how many exercises and
examples to suggest, the difficulty of exercises, in which sequence items will
be shown, which feedback will be given.

There are the classical pedagogical strategies  Didactic,
Socratic, Inquiry, and Discovery learning \cite{Keegan93}. 
Another dimension for devising strategies is its learning goal-level (K/C/A/T).
For instance, \cite{ShuteSMART}  suggests different reactions  according to
different targeted mastery levels.

In the tutoring literature, mostly  strategies for {\em local} feedback are
described. And even if these strategies are relatively general, the realization
of a strategy is pretty specific for a particular learning area so far. For 
instance,
the tutor system Miss Lindquist \cite{HeffernanThesis} has four strategies for
teaching problem solving: concrete-articulation, decompose-into-subgoals,
translate-representation from text into computation and abstract formula and
vice versa, and present model solutions, The concrete-articulation strategy for
solving algebra word problems consists of the rules:
\begin{itemize}
\item if wrong answer, then present simpler  exercise about a sub-formula
  which requires a number as the answer
\item then ask to provide the computation steps
\item then present variablized steps
\item then present full formula.
\end{itemize}

In {\activemath}, the first  strategies will be related to the four
learning goal-levels: knowledge, comprehension, application, and transfer.
%
The strategy formed by the rules suggested in \S\ref{sec:suggestors} is a
strategy for A-level learning
%, with no consideration of the field and verbosity of the learner, and 
with a relatively strong guidance.
\ednote{Merke: wie wird field und ?verbosity uebernommen? Bisher noch gar nicht
  aber Zukunft in der course 
generation durch hartes Kopieren einiger Regeln in die Subclass NBdes course generator}

%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Empirical Questions}
%%%%%%%%%%%%%%%%%%%%%%%%
Most importantly, the diagnosis and suggestion mechanisms have to be evaluated
empirically with actual users. Then, the mechanisms will be improved according
to the test results.

For some of the specifications there exist no empirical basis yet. For instance,
the following questions could be investigated empirically:
\begin{itemize}
\item what is the individual optimal reading time for instructional items?
\item how can motivational features diagnosed reliably?
\item when to provide immediate and when delayed feedback?
\item when to repeat a typical example?
\item when to try an exercise again?
\item when to present a counter example?
\item when to ask for reasons for a step (stimulate meta reasoning)?
\item  how should the global  suggestion rules be rated? 
\item should the correct solution be shown in the global feedback?
\item when to show seen and when show unseen examples and exercises
% \item for local feedback:
%    when to show  alternative solutions (system initiative)? 
%    when just ask for repetition  (user initiative)?
%   when to correct user's actions (system initiative)?
\end{itemize}

\paragraph{Design}
The design of the multi-modal suggestions has to be investigated. Among others,
design questions comprise the following: how to personalize a suggestion, which
number and position of feedback and suggestion windows is appropriate for
learning? Should the content be presented in a separate window? Should an
example solution or a correct exercise solution be shown next to a faulty
solution of a failed exercise?
Moreover, we need to experiment with the buttons in order to find out about
their usability.

\paragraph{Diagnoses for Reading Speed}
Apart from recognizing the optimal individual average reading speed there will
be more subtle differences between reading different types of instructional
elements. For instance, reading time for examples should include time for
self-explanation for a better comprehension.  Note also that the reading speed
for formulas may individually differ from the reading time for normal text.

{\activemath} can serve as a tool for controled experiments determining
dependencies of user characteristics and for benefits of many other
instructional decisions.
%%%%%%%%%%%%%%%%%%%%%%%%
\section{Conclusion}
%%%%%%%%%%%%%%%%%%%%%%%%
The web-based learning environment {\activemath} presents content, worked-out
examples and exercises to the student rather than exercises only or examples
only. This presentation may include incomplete examples, elaborations and
examples with built-in questions, etc.  
An on-line demo of {\activemath} is available at {\tt
  http://www.activemath.org}. 

This article mainly describes our research on global feedback in {\activemath}
and beyond. It also includes the distinction between local and global feedback as
well as the separation of diagnosis and global suggestion mechanisms by an
architecture with two blackboards. The suggestion mechanism is implemented in
Java.

\comment{
%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Related Work}
%%%%%%%%%%%%%%%%%%%%%%%%
888

compare with Corbett's knowledge tracing. There, feedback is hints (local)= and
choose next problem (global)??? wirklich ???.
But DBB is more flexible since not hard coded and different suggestions and
mechanisms can use the diagnoses (also TEMPORARY diagnoses on DBB)

relate to evidence centered diagnosis and feedback
\cite{Mislevy,Steinberg,Messick92} that relates student model(claims,goals,um)
to evidences and this to tasks. diagnosis rules for mastered, misconceptions,
bugs.
But we don't need  model of expert solutions to assess...

look at Malcolm Bauer (CISCO) in WSProceedings formative feedback = local(task
level), summative feedback = global? 4-process model \cite{Mislevy}. However 
our
global diagnoses are not all stired in UM but some just temporarily. Is SIE
similar to external systems?
}
%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Acknowledgement}
%%%%%%%%%%%%%%%%%%%%%%%%
We thank the {\activemath} group, in particular, Carsten Ullich and
 and Sabine Hauser for the involved discussions as well as
Bernhard Jacobs for his supply of empirical knowledge.

\bibliographystyle{plain}
\begin{thebibliography}{10}

\bibitem{AlevenKoedingerITS00}
V.~Aleven and K.R. Koedinger.
\newblock Limitations of student control: Do student know when they need help?
\newblock In G.~Gauthier, C.~Frasson, and K.~VanLehn, editors, {\em
  International Conference on Intelligent Tutoring Systems, ITS 2000}, pages
  292--303. Springer-Verlag, 2000.

\bibitem{AlevenKoedingerCross99}
V.~Aleven, K.R. Koedinger, and K.~Cross.
\newblock Tutoring answer explanation fosters learning with understanding.
\newblock In S.P. Lajoie and M.~Vivet, editors, {\em Artificial Intelligence in
  Education}, pages 199--206. IOS Press, 1999.

\bibitem{Mislevy99}
R.G. Almond, L.S. Steinberg, and R.J. Mislevy.
\newblock A sample assessment using the four process framework.
\newblock Educational Testing Service, 1999.
\newblock http://www.education.umd.edu/EDMS/mislevy/papers/FourProcess.pdf.

\bibitem{Arshad90}
F.~Arshad.
\newblock {\em The Design of Knowledge-Based Advisors for Learning}.
\newblock PhD thesis, School of Education, University of Leeds, 1990.

\bibitem{PACTtutors97}
A.T.Corbett, K.R. Koedinger, and J.R. Anderson.
\newblock Intelligent tutoring systems.
\newblock In Landauer Helander, M.~G., T.K., and P.V. Prabhu, editors, {\em
  Handbook of Human-Computer Interaction}, pages 849--874. Elsevier Science,
  The Netherlands, 1997.

\bibitem{Bloom}
B.S. Bloom, editor.
\newblock {\em Taxonomy of Educational Objectives: The Classification of
  Educational Goals: Handbook I, Cognitive Domain}.
\newblock Longmans, Green, Totonto, 1956.

\bibitem{its02}
J.~Buedenbender, E.~Andres, Adrian Frischauf, G.~Goguadze, P.~Libbrecht,
  E.~Melis, and C.~Ullrich.
\newblock Using computer algebra systems as cognitive tools.
\newblock In S.A. Cerri, G.~Gouarderes, and F.~Paraguacu, editors, {\em 6th
  International Conference on Intelligent Tutor Systems (ITS-2002)}, number
  2363 in Lecture Notes in Computer Science, pages 802--810. Springer-Verlag,
  2002.

\bibitem{ConatiAIED01}
A.~Bunt, C.~Conati, M.~Huggett, and Kasia Muldner.
\newblock On improving the effectiveness of open learning environments through
  tailored support for exploration.
\newblock In J.D. Moore, C.L. Redfield, and W.L. Johnson, editors, {\em
  International Conference on Artificial Intelligence and Education}, pages
  365--376. IOS Press, 2001.

\bibitem{BrownBurton82}
R.R. Burton and J.S. Brown.
\newblock An investigation of computer coaching for informal learning
  activities.
\newblock In D.~Sleeman and J.~Brown, editors, {\em Intelligent Tutor Systems}.
  Academic Press, 1982.

\bibitem{ButlerWinne95}
D.~Butler and P.~Winne.
\newblock Feedback and self-regulated learning: A theoretical synthesis.
\newblock {\em Review of Educational Research}, 65(3):245--281, 1995.

\bibitem{ChiDeLeeuwChiuLavancher94}
M.T.H. Chi, N.~De Leeuw, M.~Chiu, and C.~Lavancher.
\newblock Eliciting self-explanations improves understanding.
\newblock {\em Cognitive Science}, 18:439--477, 1994.

\bibitem{ChiDialog01}
M.T.H. Chi, S.A. Siler, H.~Jeong, T.~Yamauchi, and R.G. Hausmann.
\newblock Learning from tutoring.
\newblock {\em Cognitive Science}, 25:471--533, 2001.

\bibitem{ConatiAIED99}
C.~Conati and K.~VanLehn.
\newblock Teaching meta-cognitive skills: Implementation and evaluation of a
  tutoring system to guide self-explanation while learning from examples.
\newblock In S.P. Lajoie and M.~Vivet, editors, {\em Artificial Intelligence in
  Education}, pages 297--304. IOS Press, 1999.

\bibitem{soldato:aied89}
T.~del Soldato.
\newblock Detecting and reacting to the learner's motivational state.
\newblock In D.~Biermann, J.~Breuker, and J.~Sandberg, editors, {\em
  Proceedings of AIED 89}, pages 567--574, Amsterdam, Springfield, Tokyo, 1989.
  IOS.

\bibitem{Jess}
E.~Friedman-Hill.
\newblock {\em Jess, the Java Expert System Shell}, 2000.
\newblock {\tt http:/herzberg.ca.sandia.gov/jess/}.

\bibitem{HeffernanThesis}
N.T. Heffernan.
\newblock {\em Intelligent Tutoring Systems have Forgotten the Tutor: Adding a
  Cognitive Model of Human Tutors}.
\newblock PhD thesis, School of Computer Science, Carnegie Mellon University,
  2001.

\bibitem{Keegan93}
M.~Keegan.
\newblock Optimizing the instructional moment.
\newblock {\em Educational Technology}, pages 17--22, April 1993.

\bibitem{MandlGruberRenkl97}
H.~Mandl, H.~Gruber, and A.~Renkl.
\newblock {\em {Enzyklop{\"a}die der Psychologie}}, volume~4, chapter {Lernen
  und Lehren mit dem Computer}, pages 436--467.
\newblock Hogrefe, 1997.

\bibitem{MayoMitrovichMcKenzie00}
M.~Mayo, A.~Mitrovich, and J.~McKenzie.
\newblock An intelligent tutoring system for capitalisation and punctuation.
\newblock In M.~Kinshuk, C.~Jesshope, and T.~Okamoto, editors, {\em Adavanced
  Learning Technology: Design and Development Issues}, pages 151--154. IEEE
  Computer Science, 2000.

\bibitem{activemathAIEDJ01}
E.~Melis, J.~Buedenbender~E. Andres, Adrian Frischauf, G.~Goguadse,
  P.~Libbrecht, M.~Pollet, and C.~Ullrich.
\newblock {\sc ActiveMath}: A generic and adaptive web-based learning
  environment.
\newblock {\em Artificial Intelligence and Education}, 12(4):385--407, winter
  2001.

\bibitem{MitrovichOhlsson99}
A.~Mitrovich and S.~Ohlsson.
\newblock Evaluationof a constraint-based tutor for database language.
\newblock {\em International Journal on Artificial Intelligence in Education},
  10(3-4):238--256, 1999.

\bibitem{MurrayEtal00}
T.~Murray, J.~Piemonte, S.~Khan, T.~Shen, and C.~Condit.
\newblock Evaluating the need for intelligence in an adaptive hypermedia
  system.
\newblock In G.~Gauthier, Claude Frasson, and K.~VanLehn, editors, {\em
  Intelligent Tutoring Systems, Proceedings of the 5th International Conference
  ITS-2000}, volume 1839 of {\em LNCS}, pages 373--382. Springer-Verlag, 2000.

\bibitem{ReiserEtal94}
B.J. Reiser, W.A. Copen, M.~Ranney, A.~Hamid, and D.Y. Kimberg.
\newblock Cognitive and motivational consequences of tutoring and discovery
  learning.
\newblock Technical Report~54, The Institute for the Learning Sciences,
  Northwestern University, 1994.

\bibitem{Renkl01}
A.~Renkl.
\newblock Worked-out examples: Instructional explanations support learning by
  self-explanations.
\newblock {\em Learning and Instruction}, 2001.

\bibitem{ShuteSMART}
V.J. Shute.
\newblock {SMART}: Student modeling approach for responsive tutoring.
\newblock {\em User Modeling and User-Adapted Interaction}, 5:1--44, 1995.

\bibitem{DelSoldato-its92}
Teresa~Del Soldato.
\newblock Detecting and reacting to the learner's motivational state.
\newblock In , editor, {\em Proceedings of the International Conference on
  Intelligent Tutoring Systems}, volume 608 of {\em Lecture Notes in Computer
  Science}, pages 567--574. Springer, 1992.

\bibitem{Stark00b}
R.~Stark.
\newblock {Instruktionale Effekte beim Lernen mit unvollst{\"a}ndigen
  L{\"o}sungen}.
\newblock Forschungsberichte 117, Ludwigs-Maximilian-Universit{\"a}t,
  M{\"u}nchen, Lehrstuhl f{\"u}r Empirische P{\"a}dagogik und P{\"a}dagogische
  Psychologie, 2000.

\bibitem{eLearn2002}
C.~Ullrich and E.~Melis.
\newblock The poor man's eyetracker in {\activemath}.
\newblock In {\em Proceedings of the World Conference on E-Learning in
  Corporate, Government, Healthcare, and Higher Education (eLearn-2002)}.

\bibitem{VanLehn01}
K.~VanLehn.
\newblock Bayesian student modeling, user interfaces and feedback: A
  sensitivity analysis.
\newblock {\em International Journal of Artificial Intelligence in Education},
  12:.., 2001.

\bibitem{Vygotsky78}
L.~Vygotsky.
\newblock {\em The Development of Higher Psychological Processes}.
\newblock Harvard University Press, Cambridge, 1978.

\bibitem{SciWise99}
B.Y. White and T.A. Shimoda.
\newblock Enabling students to construct theories of collaborative inquiry and
  reflective learning: Computer support for metacognitive development.
\newblock {\em International Journal of Artificial Intelligence in Education},
  10:151--182, 1999.

\bibitem{Woolf84}
B.P. Woolf.
\newblock {\em Context-Depending Planning in a Machine Tutor}.
\newblock PhD thesis, Department of Computer and Information Science,
  University of Massachusets, 1984.

\end{thebibliography}

%\bibliography{/opt/local/home/erica/schreiberei/bibs/general,/opt/local/home/erica/schreiberei/bibs% /mypublications-rev,/opt/local/home/erica/schreiberei/bibs/its}
% \bibliography{/home/melis/schreiberei/bibs/general,/home/melis/schreiberei/bibs/mypublications-rev,/home/melis/schreiberei/bibs/its}c
\end{document}


%%% Local Variables: 
%%% mode: latex
%%% TeX-master: t
%%% End: 




