Engineering the Learning Process

Leveraging Science and Technology for Effective Instruction

There is currently unprecedented interest in the potential of technology to transform learning. This buzz around technology and learning is especially loud in higher education, where pundits, entrepreneurs and academics offer outspoken predictions that technology-enhanced learning (TEL) will productively disrupt the sector by addressing long-standing structural issues and the dual challenges of cost and attainment.


The massive online open course (MOOC) approach, which uses technology to scale lecture and teaching efforts, has been a particular focus of attention in many fields, including biomedical data science. But while this video-focused approach has been successful in expanding access to educational resources, its impact on learning is less clear. The accelerating pace of advances in bioinformatics demands new, more effective approaches to training and education for students and experienced practitioners alike. How can technology be used to meet this need?


One approach, called learning engineering, entails the use of learning research and the affordances of technology to design and deliver innovative, instrumented educational practices with demonstrated and measurable outcomes. This approach has proven quite successful in a variety of contexts over the last few decades, resulting in accelerated learning1, higher outcome achievement2, improved retention3, and higher-order skills attainment, all across diverse and often vulnerable learner populations.4 The data from these innovations are also used to refine and advance theory, fueling a virtuous cycle of research and practice.


This article highlights seven core practices that characterize how the learning engineering approach can be applied to develop technology-enhanced learning tools and courses in bioinformatics, biomedical data science and related fields. Although some of these ideas may seem obvious, putting them into practice requires time and effort as well as dedication to the goal of using technology to be a more effective teacher.


1) Use evidence-based design: You are a scientist. So let learning science inform the design of your learning environment. Despite evidence that research-based instruction supports robust learning more effectively than instruction guided by intuition, many faculty continue to design online courses using their personal sense of what works. Don’t fall into that trap. Spend time with the learning engineering literature (the Global Learning Center whitepaper referenced below makes a good start) and get advice from learning engineering experts. Guidance and support in evidence-based instruction can often be found in your own institution’s center for teaching excellence, but resources are also available from programs such as the Simon Initiative at Carnegie Mellon; the iAMSTEM program at the University of California5; and the Kirwan Center for Academic Innovation at the University System of Maryland. A new report from MIT’s Online Education Policy Initiative also provides evidence and advocacy for the learning engineering approach.


Learning engineering produces a data-driven virtuous cycle of learning research and innovative educational practice, causing demonstrably better learning outcomes for students from any background.2) Begin with a model: As you design, develop an explicit model of the learning that you are supporting. What are the objectives and related skills that learners need to achieve? A useful model will describe these elements in a measurable way, and will map them to the activities and assessments that you develop. In a statistics course, for example, a simple model might first explicate the learning objective “Relate measures of center and spread to the shape of the distribution, and choose the appropriate measures in different contexts,” with sub-skills that could include “Compute median” and “Identify outlier.” These skills would then be mapped to learning activities and assessments, creating a model that can support your design and data-driven analysis. is an example of applying this kind of modeling in the bioinformatics arena (See Skills Upgrades story in this issue).


3) Focus on learn-by-doing activities: Design active learning experiences, embed practice opportunities in the problem-solving context, and provide learners with targeted feedback that speaks to misconceptions. While this learn-by-doing approach can be expensive, the potential pay-off for your learners is tremendous. A recent study6 found relatively simple learn-by-doing activities required no additional time investment by the learner but were six times more effective in supporting learning than the readings and video lectures that are central to many MOOCs.* In the bioinformatics and biomedical data science fields, some MOOC designers are finding creative ways to incorporate learn-by-doing activities into online learning. (See the work by Pavel Pevzner, Brian Caffo and Rafael Irizarry described in the Skills Upgrades story in this issue).


4) Continuously research, iterate and improve instruction: As with most fields of study, the science of learning changes over time. By revising courses to address developments not previously studied, educational materials grow increasingly effective and robust over time. And again, it’s scientific: Learning engineers treat the development of instructional activities as a hypothesis on how learners will best achieve a given learning outcome; capture data from learners’ interactions to evaluate the hypothesis; and then take the essential step of closing the loop by using this new information to refine the theory and improve the learning activities. The Carnegie Mellon DataLab offers some tools (links below) to help teachers cycle through this process.


5) Capture rich learning data: Rich learning data fuels effective feedback to learners and educators, drives iterative improvement, and supports advances in learning research. Most online courses only track which pages students visit in which order (so-called click-stream data) rather than meaningful interactions with the material. Activities should be designed to produce useful data. For example, multiple-choice problems should include incorrect answers designed to highlight learner misconceptions (rather than serving as mere distracters), and offer an immediate means to correct those misconceptions. Also, students should be given multiple opportunities to solve open-ended exercises in order to generate data about students’ errors, thereby highlighting conceptual problems and demonstrating ways the course should be improved. These activities allow students to engage meaningfully with authentic problems in ways that provide observable information about the learner’s knowledge state in relation to specific learning objectives.


6) Choose high quality platforms and tools: Avoid reinventing the wheel. Locate and use existing materials that were designed using learning science research; and choose tools and platforms with care to ensure that they provide features that allow you to accomplish the tasks described above. For example, the cognitive tutor authoring toolkit (CTAT) out of Carnegie Mellon can help educators develop valuable online educational tools. In addition, extensive open educational resources (OERs) are available in many domains that are effective in their own right and can serve as a strong foundation for continued improvement and experimentation. Indeed, several are currently being created for the bioinformatics arena, including and, which are both described in the Skills Upgrades story in this issue, as well as OERs under development through Oregon Health and Science University with funding from the National Institutes of Health Big Data to Knowledge program.


7) Bring together multi-disciplinary teams: Online instruction is too often developed by a single educator working alone. The educator might have disciplinary expertise and classroom experience, but producing effective TEL resources requires many other talents including expertise in design, instruction, technology, learning science and human-computer interaction. 


1 Lovett 2008

2 Bowen, 2012; Evans 2008; Griffiths 2014; Koedinger 2015

3 Schunn 2009

4 Kaufman 2013; Ryan 2015

5 Perez-Pena 2014

6 Koedinger 2015



Norman Bier is director of the Open Learning Initiative (OLI) and the executive director of the Simon Initiative at Carnegie Mellon University.



Simon Initiative ( as well as its DataLab ( and cognitive tutor authoring tools (
LearnLab ( – the Pittsburgh Science of Learning Center – offers tools and research.
The Eberly Center for Teaching Excellence and Educational Innovation ( provides additional materials to support evidence-based design.
The Open Learning Initiative ( offers online learning environments that exemplify the learning engineering approach.
A recent white paper by the Global Learning Council titled Technology-Enhanced Learning: Best Practices and Data Sharing in Higher Education (2015) is available from the Council's web site (



Bowen, W. G., Chingos, M. M., Lack, K. A. and Nygren, T. I. (2014), Interactive Learning Online at Public Universities: Evidence from a Six-Campus Randomized Trial. J. Pol. Anal. Manage., 33: 94­111. doi: 10.1002/pam.21728


Evans, K. L., Leinhardt, G., & Yaron, D. (2008). Learning stoichiometry: A comparison of text and multimedia formats. Chemistry Education: Research and Practice, 9, 208-218.


Griffiths, R., Chingos, M., Mulhern, C.,  & Spies, R.. (2014). Interactive Online Learning on Campus: Testing MOOCs and Other Platforms in Hybrid Formats in the University System of Maryland. New York: ITHAKA S+R.


Kaufman, J.; Ryan, R.; Thille, C. and Bier, N. (2013) Open Learning Initiative Courses in Community Colleges: Evidence on Use and Effectiveness. Carnegie Mellon University, Pittsburgh, PA.


Koedinger, K., Kim, J., Zhuxin, J., McLaughlin, E., and Bier, N. (2015). Learning is Not a Spectator Sport: Doing is Better than Watching for Learning from a MOOC, Proceedings of the ACM Conference on Learning @ Scale.


Lovett, M., Meyer, O., & Thille, C. (2008). The Open Learning Initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education.


Perez-Pena, R. (2014). Colleges Reinvent Classes to Keep More Students in Science. The New York Times. 26 December, 2014.


Ryan, S., Kaufman, J., Greenhouse, J., She  R., & Shi J.(2015): The effectiveness of blended online learning courses at the community college level, Community College Journal of Research and Practice, DOI: 10.1080/10668926.2015.1044584


Schunn, C. D. & Patchan, M. M. (2009). An Evaluation of Accelerated Learning in the CMU Open Learning Initiative Course “Logic & Proofs”. Technical report by Learning Research and Development Center, University of Pittsburgh., Pittsburgh, PA.


Simon, H.A. (1967). The job of a college president. Educational Record, 48, 68-78.


Technology-Enhanced Learning: Best Practices and Data Sharing in Higher Education. (2015) White paper prepared for the Global Learning Council.


Thille, C. and Smith, J. (2004). The Open Learning Initiative: Cognitively Informed E-Learning, The Observatory on Borderless Higher Education.



All submitted comments are reviewed, so it may be a few days before your comment appears on the site.

Post new comment

The content of this field is kept private and will not be shown publicly.
All submitted comments are reviewed, so it may be a few days before your comment appears on the site. This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Enter the characters shown in the image.