15.1
Separating product quality understanding and transfer
Evaluation has at least three levels. Production quality concerns accuracy, clarity, accessibility, and maintainability. Understanding concerns explanation of relationships, conditions, and errors. Transfer concerns selecting or revising a representation in a new situation. The levels relate but cannot substitute for one another. An excellent AI-generated graphic may be submitted by someone with limited understanding, while a visually rough sketch may encode accurate and insightful structure. Assessment should separate them.
An analytic rubric can describe observable performance. Semantic accuracy checks important nodes and edges against evidence. Reasoning completeness checks conditions, exceptions, and counterexamples. Traceability checks whether important claims can be located. Readability checks encoding, hierarchy, and text. Transfer checks revisions under changed conditions. Appearance belongs within communication, but should not outweigh evidence and reasoning. Examples should include attractive errors and plain correct work.
Understanding should also survive beyond the finished artifact. An instructor can select an edge and ask for its source, role, and consequences if removed, or provide a new source passage for immediate revision. A short defense reveals ownership of the reasoning. Large classes can combine sampled defenses with independent short assessments to control cost.
15.2
An experiment on AI generation and feedback
A study of AI-supported visual thinking can compare four conditions: studying an instructor-provided diagram, drawing independently, explaining and revising an AI-generated diagram, and drawing first followed by question-based AI feedback. Content, approximate total time, and assessment remain comparable. Preregistered hypotheses might predict faster production from direct generation and stronger relational explanation from feedback. These remain hypotheses rather than results announced in advance.
A primary outcome could be relational inference or transfer in a new setting. Secondary outcomes include retention, time, cognitive load, graphic accuracy, and confidence calibration. Immediate and delayed testing address different questions. An unaided transfer task is particularly important because the intended outcome concerns human capability. Product quality during AI use alone cannot distinguish learning from tool substitution. Actual revisions should also be recorded.
Random allocation addresses some confounding, but classes, instructors, prior knowledge, and collaboration still matter. Class-level implementation needs corresponding limitations and analysis. Sample size should follow the smallest meaningful difference, measurement reliability, and clustering through a power analysis. There is no context-free universal sample size, and an exploratory classroom activity should not be presented as a completed randomized trial.
Conceptual design proposed in this report
Figures retain bilingual labels; the surrounding text explains them. Open for a larger view.15.3
Measuring processes and mechanisms
Process evidence can explain success or failure. The order of source reading, node revision, hint requests, and feedback review may show whether integration occurred. Collection should serve a clear question with appropriate information and permissions rather than gather behavior indefinitely. For routine course improvement, a few important versions and explanations may suffice without elaborate monitoring.
Mechanistic inference requires caution. A correlation between revisions and achievement may mean that stronger learners revise more, that revision supports learning, or both. Testing whether feedback affects transfer through relational accuracy requires suitable design and assumptions. Rich logs do not automatically provide causal evidence. Reliable measurement should precede complicated explanatory models.
Confidence calibration is also useful. Learners estimate certainty after a judgment, then compare it with accuracy. AI may make work look complete and raise confidence without a matching accuracy gain. The gap can reveal more than a general preference survey. The ability to identify an uncertain relationship should count as part of professional judgment.
15.4
Analysis reporting and reproduction
Reports should present estimates, intervals, and task descriptions rather than significance labels alone. A larger effect for learners with less prior knowledge should be identified as planned or exploratory. Graphics should show suitable distributions or individual change instead of treating the mean as everyone's experience. A nonsignificant result does not establish equivalence without considering precision and design.
A reproducibility package can contain anonymized tasks, instructional materials, rubrics, analytical code, chart data, and versions. Restricted data need a stated limitation and possible substitutes. Model names, dates, prompt templates, and consequential parameters may affect outcomes and should be recorded, while irrelevant private conversations need not be disclosed. Bilingual tasks require checks that translation has not changed difficulty.
Findings should guide the next iteration. Accurate but slow feedback may need shortening. Faster production with weaker transfer may require more independent construction. Some tasks may not benefit from AI. The aim is to identify conditions and mechanisms rather than establish universal value for new tools. Visible processes support this revisable educational knowledge.
| Dimension | Basic | Stronger |
|---|---|---|
| Semantic accuracy | Main nodes identifiable | Relations match evidence |
| Reasoning | Normal route coherent | Exceptions and counterexamples handled |
| Evidence | General sources supplied | Important elements traceable |
| Readability | Text and structure legible | Task-serving accessible encoding |
| Revision | Can change appearance | Explains conceptual effects |
| Transfer | Reproduces learned case | Adapts to new conditions |