Introduction and Overview

Picture this. An AV-8B Harrier avionicsman aboard the aircraft carrier USS Peleliu on deployment to Afghanistan is called in to troubleshoot a problem with an aircraft’s LITENING II Targeting Pod (TPOD). The pod gives the Harrier a new capability to autonomously deliver precision guided munitions, increase day and night target acquisition and improve low-level night flight capabilities, 24 hours a day and in adverse weather conditions. It is a new piece of equipment introduced to the Harrier fleet in 2001. The avionicsman has never seen a pod before this one and, as training was not available on the pod in his fleet squadron before he was deployed, he has no idea how to repair it. Not to worry. The Marine retrieves his personal training delivery device and has the Learning Management System (LMS) run a query for “LITENING II Targeting Pod.” Seconds later, his search results return a link to a high-fidelity simulation training device—a self-contained piece of interactive courseware developed for the NAMTRA MARUNIT aboard MCAS Cherry Point, NC, designed to train new avionicsmen on troubleshooting and repair of the pod. The Marine selects the link and instantly the learning object is downloaded from its repository and is available on his machine. He selects the symptoms exhibited by his pod and quickly goes through the troubleshooting and repair techniques to correct the discrepancy. Armed with his new knowledge and a technical publication, within minutes he identifies the problem with his pod—a faulty Weapons Replaceable Assembly (WRA) that is designated for manufacturer repair. He pulls the WRA, packages it for return to the manufacturer and reloads a new WRA. The pod is functional within the hour.

Now picture this. An Instructional Systems Designer (ISD) works for an agency contracted to design and develop computer-based training (CBT) for the Marine Corps. The ISD is storyboarding a course designed to teach engine test cell operators how to troubleshoot and repair T58 helicopter engines on an engine test system. The third lesson in this course is to be an overview of basic electricity. In this lesson, the test cell operator would receive instruction on the types of electricity, flow, voltage, and use of a multimeter. The ISD, with guidance from a Subject Matter Expert (SME), is aware that knowledge of electricity and the skill in the use of multimeters are not unique to engine test cell operators. Rather, there are many Military Occupational Skills (MOS) that require such foundational knowledge. Before beginning to storyboard this lesson, the ISD pulls up a browser window on his computer and goes to a bookmarked web site, a search engine for a content repository hosted on a Department of Defense (DoD) server. He types in “basic electricity” and receives 1243 hits. By narrowing his search a bit and browsing for a few minutes, the ISD locates an existing lesson on basic electricity that covers the same material he needs to cover in his test cell operator lesson. He downloads it to his system and extracts the data to his storyboard. By adding a short introduction and summary, the ISD is able to integrate the entire existing basic electricity lesson into his course. His course is on the way to being complete and he hasn’t even finished his first cup of coffee for the morning.

These scenarios represent a utopic view of how learning, training, and training design and development will take place in a transformed United States military. Learning is individualized, just-in-time (JIT) and just enough. A single deployed Marine is able to pull just the training material he needs, no more and no less, when he needs it. Training design and development is modular in nature and incorporates reusable content for speed and efficiency of production. In both scenarios, accessing the proper material took just a few seconds. The training already existed; no one had to create a complex simulation especially to meet this Marine’s learning need and the ISD was able to avoid reinventing the wheel. Training that’s better, faster, and cheaper: such is the future of military training.

Achievement of such a vision requires more than a wish list, funding, and an Internet connection. It requires a strategy. A strategy is a careful plan or method (Merriam-Webster, 1990.) Approaching the accomplishment of this goal using a strategic model will help the Department of Defense (DoD) better understand its current situation and ensure that any action taken is in support of overall organizational goals (Driscoll, 2002, Rothwell & Kazanas, 1998). Recognizing the advantages of a strategic approach in making these scenarios a reality, the DoD has launched the Advanced Distributed Learning (ADL) Initiative.

This paper seeks to evaluate the efforts of the Department of Defense in implementing this e-learning strategy. The ADL Initiative is what I would call a “back end” effort. In other words, it affects and touches the DoD learner end-user only indirectly. Rather, the ADL Initiative is designed to enable content developers, be they within the DoD or government contractors, to more effectively and efficiently provide quality e-learning material to the DoD learner end-user. This distinction is better defined and explained prior to evaluating the DoD initiative. E-learning models are examined and a single strategic model is selected against which to evaluate the effort. The ADL Initiative is then evaluated along six key strategic dimensions against specific criteria. The evaluation is performed from the point-of-view of the author, an ISD employed by a government contractor experienced in developing e-learning content for the Marine Corps.

The Problem

On March 1, 2002, the Director of Readiness and Training Policy and Programs in the Office of the Undersecretary of Defense for Personnel and Readiness issued the “Strategic Plan for Transforming DoD Training” (Kaufman & Svitak, 2002, p. 2). This plan elevates the importance of training in the military and “seeks to provide combatant commanders with troops, units and leaders that are better prepared to work with different organizations and to be flexible enough to meet the unknown contingencies of the 21^st century” (Kaufman & Svitak, 2002, p. 2). By emphasizing the “joint” in joint operations, the plan seeks to capture key operational innovations that occur when all four branches of the services come together in situations like that in Afghanistan. Refining, then capturing processes that work during joint operations and learning from them will improve the ability of the Army, Navy, Air Force and Marines to work together for a stronger military. By October 2003, a list of joint core competencies will be developed that will drive service, component and staff training requirements.

Key to the plan is the implementation of a vast, worldwide training network that connects troops, units and commands throughout the military and allows access to real and virtual training whenever and wherever it’s needed. While the plan emphasizes joint training, still, it recognizes the need for each individual service to master its own core competencies. The branches are already making use of the Internet to meet training needs and have been for some time. The Navy, for example, developed an online training portal in 2001. Navy E-Learning offers up to 2000 courses to over 1.2 million retired and current Navy and Marine Corps personnel, civilians and dependents (Harris, 2002).

Clearly with the implementation of this plan, military training will undergo a transformation. Training is no longer a stand-alone application, but is an integral part of the development of a serviceman or woman, one equipped and able to respond decisively to any type of challenge. “But the plan isn’t an invitation to spend large sums of money...it’s a summons to use current resources in a more clever fashion...” (Harris, 2002, p. 2)

A mandated joint training network to meet rapidly changing training needs; an enormous, geographically dispersed audience; a need to keep spending in check—these requirements make e-learning seem a custom-made solution for the DoD. E-learning is scalable and allows any number of learners to access learning content at any place and time. Online learning content is easily updated and distributed to ensure the most up-to-date content is provided to learners. The vast network of the World Wide Web connects learners to each other and to content Subject Matter Experts for real-time or delayed collaboration and conversation. And by accessing learning content online, learners and the DoD avoid the cost of travel, time investment, and facilities required for on-site training delivery (Beer, 2000).

And yet, a survey of people and organizations responsible for teaching as many as 30,000 courses annually within the DoD found a less than 5% insertion rate of technology into those courses. Investigation into these results revealed that respondents were reluctant to implement technology because the solutions available were platform dependent, version specific, monolithic in construction, would not operate with other systems, and were exceedingly expensive (Mark Oehlert, personal communication, September 5, 2002). Structural and organizational issues such as personnel policies and training procedures are also obstacles (SCORM, 2001). Clearly, if e-learning is to be a viable solution within the DoD and just-in-time, anywhere, anytime learning is to become a reality, things need to change.

E-Learning: A Theoretical Framework for Success

The Knowledge Economy and Training

The transformation of the military isn’t the only change game in town. Beyond the gates of the military bases and walls of the DoD, the entire world of work is undergoing a transformation of its own. Technological advances in computing and networking have had a mind boggling effect on how organizations do business (Marquardt & Kearsley, 1999). With the development and explosive growth of the Internet, notably a Department of Defense initiative originally designed as a communication tool in a post-nuclear America, the Industrial Age has been effectively replaced by the Knowledge Age. Brains have surpassed brawn as a desirable work skill. Information and knowledge have “become the preeminent economic resource—more important than raw material; more important, often, than money” (Stewart, 1997, p.6).

The proliferation of computers into the workplace has effectively decreased the need for unskilled labor and accelerated the demand for skilled, so-called “knowledge workers,” high-level employees who apply theoretical and analytical knowledge, acquired through formal education, to developing new products or services (Drucker, 1994). Ironically, in this services driven economy in which skilled labor is a hot commodity, resources for such workers are sorely limited. While computer related positions have grown steadily over the past decade, the number of students graduating with degrees in computer science declined rapidly from 1989 to 1997. While this second statistic saw a reversal in trend in 2000, the effects of the previous ten years are hard to erase. There simply aren’t enough knowledge workers to go around (Moncarz, 2002).

This situation presents a special challenge for organizations. In order to stay competitive, businesses must continuously develop the employees they have, instilling knowledge and skills necessary to survive in an increasingly demanding technical workplace. Learning, as a result, has become an integral part of working. A study by Torraco (1999) that examined in-depth accounts of actual work activity supports this notion. “The work descriptions show that the distinction between learning and working has significantly eroded in today’s workplace. Skilled performance in the work roles described would not have developed if workers had not had the benefit of learning in the context of their work” (p. 258). Additionally, in order to resolve the increasingly complex and poorly defined problems that many employees are faced with today, workers must be innovative and go beyond scripted procedures to successfully perform their jobs (Torraco, 1999). Today’s employee then, is the epitome of a lifelong learner. He has to be, to keep up with the break-neck pace of business and technological change. Microsoft founder Bill Gates estimates that products developed by his company are obsolete within 3 years; it isn’t a leap of logic to presume that the skills needed to operate those tools for example, require update at a same or similar rate (Gates, 2000). The lines between working and learning are increasingly blurred; while a knowledge worker works, he must continually be learning. In perhaps no other part of an organization is this change felt more than in the training function (Tobin, 1998).

Traditionally, the training function has employed an academic research model as its basic paradigm. Essentially, this means the emphasis of training has been on learning theory, instructional design methodologies, and training methods. Members of the training unit are generally the experts in such theories and methodologies, and therefore, a formalized, trainer-directed approach to instructional programs is typical (Tobin, 1998). For many years this approach to employee learning seemed to work. Today, it clearly doesn’t, for several reasons. Training under this old paradigm historically has not been linked to organizational goals and initiatives. Rather, it has traditionally been approached as a solitary effort not related to corporate strategy. The focus under this old paradigm has often been on training for the sake of training (Rothwell, 1996). This focus comes from an underlying belief by the training function that training produces learning, which in turn changes behavior, resulting in improved performance. Unfortunately, this is a false and misleading assumption. These concepts are not synonymous. Training, something that is done for others, seeks to coach in a desired behavior or performance and / or increase proficiency with instruction and practice. Learning, something one does for oneself, is to increase knowledge through study or experience (O’Driscoll, 1999). Performance is about taking action to produce outcomes, results, or accomplishments (Rothwell, 1996.) While the belief that the three are transitively related is a seductive one, the training function typically has failed to produce any evidence that such a relationship exists (O’Driscoll, 1999). As a result, the training function has been vulnerable to being viewed by management as reactive rather than proactive, dispensable, or otherwise lacking significant value, and legitimately so (Rothwell, 1996). Simply, today’s organizations need workers who can perform and the training function has to find a way to serve in that capacity.

Human Performance Technology (HPT) and Knowledge Management (KM) are burgeoning fields of interest representing the training function’s call to arms in response to these developments. HPT focuses “on systematically and holistically improving present and future work results achieved by people in organizational settings” (Rothwell, 1996). While training focuses specifically on facilitating the development of employee knowledge, skills and attitudes critical for successful job performance, HPT more broadly recognizes that training is only one part of improving worker performance (Rothwell, 1996). Looking holistically at the worker as part of a system, HPT analyzes variables at three levels that affect the performance of organizations and individuals: the organizational level, the process level, and the job / performer level. Within that structure, it is only when a deficiency in knowledge, skill or attitude can be identified at the level of the individual performer that training is the appropriate solution to a performance discrepancy (Rummler, 1999). By viewing these three levels together as part of an overall system, HPT effectively links individual performance and organizational performance, tying worker goals to the goals of the business. Finally, recognizing that performance issues involve more than just deficiencies that can be effectively addressed with a training solution, HPT can more effectively solve performance problems and yield increased returns in terms of increased performance potential (Rothwell, 1996).

The fundamental sources of wealth in the Knowledge Age are knowledge and communication rather than natural resources or physical labor. Therefore, the concept of Knowledge Management is becoming more popular as companies begin to recognize the need to more effectively leverage their intellectual assets (Stewart, 1997). There are as many definitions and descriptions of what Knowledge Management actually is as there are practitioners to ask. Prusak and Davenport (1998), thought leaders in the field of Knowledge Management define KM as a fluid mix of framed experience, values, contextual information, and expert insight that provides a framework for evaluating and incorporating new experiences and information. It originates and is applied in the minds of “knowers” (p. 5). In organizations, it often becomes embedded not only in documents and repositories, but also in organizational routines, processes, practices, and norms. In other words, KM is about opening and facilitating the flow of knowledge within a company so that it can be utilized and built on to improve the performance of individuals and teams and, as a result, the organization.. KM is intimately connected to both training and HPT, as it, too, is a planned effort by an organization to facilitate employee learning and enhance performance (Noe, 1999). However it can be differentiated from training interventions in that it is not focused on solving specifically identified knowledge, skill or attitudinal deficiencies. Rather, the emphasis is on providing access to useful and relevant information to facilitate learning.

Facilitating learning, both at the individual level and the organizational level to improve performance—that is a fundamental goal of training functions within organizations in today’s knowledge-centered climate. HPT interventions—including training—and Knowledge Management efforts all support the growth of an organization into a knowledge-enabled one. “When a company learns to utilize and foster the growth of the knowledge and skills of all employees across all functions and levels, integrate learning activities into every employee’s work, encourage and reinforce all modes of learning, and align all of this learning with the company’s strategic business directions, it becomes a knowledge-enabled organization” (Tobin, 1998, p. 39). One of the ways in which many organizations today accomplish this formidable feat is through e-learning.

From Web-based Training to E-Learning

Increasingly, as learning has integrated with working, organizations have turned to technology as a solution. Now that an Internet-connected computer is a prominent fixture on the desks of most workers, training provided over the web is rather commonplace. Training magazine’s 21^st annual industry report (2002) reveals that 90% of responding organizations use the Internet, an intranet or an extranet to deliver training. Citing the catastrophic events of September 11^th, the report acknowledges a significant decrease in business and personal travel and an increased reliance on technology to compensate for that limitation (Galvin, 2002).

A quick search on the Barnes and Noble web site (2003) generates 63 titles related to designing effective web-based training. Best practices for developing web-ready courses are fairly easy to come by. Key design rules include selecting the most appropriate WBT format, facilitating asynchronous and synchronous collaboration tools, limiting enrollment in courses, providing expert facilitation, creating a nurturing and supportive learning environment, good pedagogy, quality materials, course preview, monitoring and assessment, and attention to interface usability (Beer, 2000, Driscoll, 1998, Kilby, 2001, Tinker, 2001). In spite of these guidelines, however, the proliferation of web-based training efforts has little to do with quality and actual improvement in performance. The challenge of meeting the needs of both the learner and the provider organization is often unmet in web-based training efforts, resulting in courses of questionable use and quality (Kilby, 2001).

One potential explanation for this disconnect is a return on the part of many training functions to doing what they’ve always done. As the need for speed has made the notion of just-in-time learning more appealing, the training function has relied on its traditional approaches to solving performance issues. This generally includes determining gaps in knowledge, skills or attitudes, and developing training programs as a solution (Weintraub & Martineau, 2002). With technology now an available and viable solution, that training can be churned out in the form of web-based courses. Admittedly, these courses may be well-designed and high quality. But even though the method of delivery has changed, the same problems crop up. For example, if all an employee really needs is immediate access to a list of experts within the organization on a particular topic, a well-designed lesson on creating a corporate address book in Microsoft® Outlook® isn’t going to help. Many organizations and their training functions have failed to see how the Web can facilitate organizational and individual learning beyond simply being a vehicle for delivering training. Enter e-learning.

It is not uncommon to see the terms “e-learning” and “web-based training” used interchangeably. But those that see the potential of the Internet for facilitating learning and performance beyond the delivery of training programs distinguish between to two. The Masie Center, a think tank dedicated to exploration and research on new technologies defines e-learning as “learning or training that is prepared, delivered, or managed using a variety of learning technologies and that can be deployed either locally or globally” (Masie Center, 2002). Marc Rosenberg (2001) says that “e-learning refers to the use of Internet technologies to deliver a broad array of solutions that enhance knowledge and performance” (p. 28). Margaret Driscoll (2002) updates the subtitle on the latest edition of her book to reflect an expanded view of e-learning. She begins this newest look at e-learning, not by discussing the advantages of online instruction as in her volume from 1998, but by listing the strategic and tactical advantages of WBT as a part of an e-learning strategy. In speaking to this shift in perspective, Driscoll (2002) says, “The change in terminology is much more than semantics. The breadth of the term has widened the scope for what is to be included in the training professionals’ toolbox of instructional technologies and the scope of this book…e-learning now includes technologies for tracking and managing training; applications that assist in authoring and managing content; and a host of new collaboration and knowledge management applications…” (p. 1).

Looking at e-learning from this expanded perspective is yet another paradigm shift for organizations and training functions comfortable with the use of the Internet for delivery of WBT. Some though, recognize the potential and are taking steps toward developing an e-learning strategy to facilitate just-in-time, anytime, anywhere learning. The Department of Defense is one of these. The ADL Initiative marks the Defense Department’s foray into the world of e-learning.

The ADL Initiative: A DoD Solution

In 1997, the DoD and White House Office of Science and Technology Policy (OSTP) launched the Advanced Distributed Learning (ADL) Initiative with the purpose of “ensuring access to high-quality education, training and decision aiding materials that can be tailored to individual learner needs and made available whenever and wherever they are required” (SCORM, 2001, p. 1-11).

The use of the term “distributed” in Advanced Distributed Learning alludes to an emphasis on asynchronous technologies that deliver instruction to learners who do not have to be at a specific place at a specific time to receive the instruction. The ADL Initiative differentiates this “distributed learning” from distance learning with the assertion that traditional distance learning relies on synchronous technologies such as video conferencing in which students, while separated physically from other students and an instructor, must gather at specific times and places to participate in instruction. “Advanced” refers to the long-term vision of the ADL Initiative. Specifically, it prepares for a future in which the scenario presented at the beginning of this paper are a reality—where personal training delivery devices are available to every learner and repositories store learning objects for broad distribution and use (SCORM, 2001). “Learning” in ADL demonstrates and understanding of the differentiation between training and learning.

The ADL Initiative, while charted by the Department of Defense, has some potential application to industry and academia as well as government. To promote collaboration and cooperation between key and interested parties and to foster research, development and assessment related to the initiative, the ADL Initiative is comprised of a network of three physical co-laboratories, each having a distinct and specific responsibility. The operational command post is located in Alexandria, Virginia. The ADL Co-Laboratory (Co-Lab) as it is called, coordinates communication between the three co-laboratories and addresses policies and specifications (2003). The Joint Academic Co-Lab, located in Orlando, Florida, serves the military community. It was established to promote collaborative development of ADL systems acquisitions and prototypes, primarily among the Department of Defense components and military services (2003). The Academic ADL Co-Laboratory, located in Madison, Wisconsin, represents the voice of academia. It serves as an academic partner and ADL link to test, evaluate and demonstrate ADL-compliant tools and technologies to enhance teaching and learning (2003).

The ADL Initiative has six major goals guiding its work:

1. Establish guidelines for large-scale development and implementation of efficient and effective distributed learning

2. Identify technical challenges that exceed the state-of-the-art, and initiate collaborative R&D programs to overcome those challenges

3. Share lessons learned and accelerate the development of a robust, highly diverse object-based environment for ADL

4. Establish a networked community of education and training consumers who recognize the importance of common standards

5. Identify and promote business models and economic incentives that serve consumers and providers of distributed learning, and

6. Stimulate large-scale development by organizations that share learning requirements (ADL, 2002).

The first 3 goals focus on technology, but the ADL Initiative seems to recognize that technology is just a means to an end. “The challenges in meeting the ADL mission are not then based on technology infrastructure per se. Instead, the task is to understand how to fully utilize the next generation technology infrastructure for learning anytime, anywhere” (SCORM, 2001, p. 1-12). Still, it is the first goal—that of establishing e-learning technical standards—on which the ADL Initiative has focused its efforts thus far.

Evaluating the DoD E-Learning Strategy

E-Learning Models

Evaluation of the DoD’s e-learning initiative requires a review of existing e-learning models. Not surprisingly however, models of e-learning as more than just WBT are few and far between. There is discussion though, surrounding e-learning in the literature, and guidelines if not fully developed models, are available to ensure e-learning success.

Alexander (2001) proposes a model of e-learning in the context of higher education. Based on Trigwell’s levels of influence on student learning, Alexander argues for a systems approach to e-learning and “proposes a framework for the design, development and implementation of e-learning…within higher education” (p. 240). Alexander’s model is based on student experience, teachers’ strategies, planning, and thinking, and the teaching / learning context. It supports the notion that initiatives in e-learning must encompass more than a focus on teaching strategies. Alexander postulates that several factors are integral to e-learning success in higher education: institutional vision, a technology development plan, faculty workload policies that reflect the demands of e-learning, a reliable technology network, technology support for staff and students, market research support, faculty development, and time release for faculty working in an e-learning effort. Though the support factors listed in Alexander’s model arguably have counterparts in industry, her model is specifically targeted to higher education making it less than ideal for evaluation of the DoD initiative.

In a case study of an e-learning initiative by Anheuser-Busch, Tyler (2002) looks at a competency-based approach to e-learning. The company’s Wholesaler Integrated Learner (WIL) program is described as a training and development program based on job competencies. In the creation of WIL, Anheuser-Busch completed a competency assessment to create a competency database. Employees take tests that measure work proficiency and analyzes gaps in worker abilities. The system then offers suggestions, including web-based and traditional classroom training courses, books, and on-the-job activities that the worker can select to close those gaps. This personalized approach to employee development better ensures that any action taken by the employee will be specifically targeted to his/her needs with increased results in closing proficiency gaps(Tyler, 2002). Although the WIL competency model incorporates alternative learning activities such as mentoring into its system, it is dangerously close to the more limited view of the Internet as training delivery device. And for an organization the size of the DoD, creating a comprehensive and useful job competency database would be a daunting task, to say the least. The Department of Defense is the oldest, largest, busiest and most successful company in the country, employing more than 2 million people, operating in 146 countries from over 6000 locations. The organizational structure of the DoD encompasses multiple levels, including the Departments of the Army, Navy, Air Force, the Unified Commands and the Joint Chiefs of Staff and all of the major commands and agencies falling under each (Department of Defense, 2003). Thus, establishing core competencies for the entire department would be an impracticably cost- and time-intensive process.

Other discussions of e-learning take less of an organized model approach and instead focus on identification of best practices in developing e-learning initiatives. Common to these reports are the following guidelines on e-learning initiative development:

· E-learning is more than online courses—look beyond the course paradigm,

· E-learning does not replace instructor-led training activities,

· E-learning should support learning delivered through other efforts,

· Supporting technology infrastructure is critical,

· Becoming fixated on technology is a mistake,

· Just because you build it, they may not come—e-learning is a significant change requiring a change in culture that emphasizes learning,

· Quality content is still important (Broadbent, 2002, Cohen & Payiatakis, 2002, Weaver, 2002).

It is Rosenberg (2001) who captures these elements in a single model.

Rosenberg’s Strategic Foundation for E-Learning

According to Rosenberg, taking advantage of the benefits of e-learning requires building a successful e-learning strategy that incorporates 6 key factors:

· Building an infrastructure,

· Combining online training and knowledge management,

· Building a learning architecture,

· Developing a sound business case,

· Reinventing the training organization, and

· Managing change and developing a learning culture (Rosenberg, 2001).

A sound technological infrastructure is required to support e-learning. The second factor emphasizes the importance of both instruction and information. If e-learning is to truly support learning, it must support all of the ways in which people learn. This includes providing access to training opportunities, both on and offline, as well as access to “bald” information not designed to instruct. By a learning architecture, Rosenberg means coordinating e-learning with other learning efforts taken by the organization, including traditional classroom training. For an e-learning initiative to be viable, there must be a compelling business case for its implementation. Reinvention of the training organization refers to changing the traditional mindset of many organizations from training as an isolated function within the organization to training as one part of learning a new way. Effective e-learning strategies require an organizational shift to an environment that values learning. While this seems self-evident, creating a culture that supports learning is perhaps the biggest challenge of all. It is the combination of these elements that form a “strategic foundation for e-learning” (Rosenberg, 2001, p 35).

With a professional and academic background in both instructional design and business, Rosenberg is effectively able to see beyond the blinders of the traditional training paradigm. He has a unique understanding of the business and technological drivers that are influencing changes in learning and clearly grasps the importance of aligning the goals of the training function with organizational goals (Rosenberg, 2001). His strategic foundation for e-learning, then, embodies an academic and corporate approach, applicable to an organization such as the Defense Department.

Context for Using Rosenberg’s Model

With respect to an e-learning strategy within the DoD, as in any organization or academic institution, there are multiple stakeholders. These stakeholders generally fall into one of three categories. End-user stakeholders are the participants and include the learners or employees who will actually be using the e-learning system to learn anytime, anywhere. Strategic stakeholders are the organizational leaders who provide management and financial support for the effort and may include the CEO, CIO or CTO. Operational stakeholders include managers, IT, instructors and trainers who may use the e-learning system to support or facilitate instruction, and content developers who provide instructional and informational content to the e-learning system (Broadbent, 2002).

The six guiding principles of the ADL Initiative speak to the operational group of stakeholders. This is more clearly stated on the ADL web site (2003): “The ADL Initiative is designed to accelerate large-scale development of dynamic and cost-effective learning software and systems to stimulate an efficient market for these products in order to meet the education and training needs of the Military Services and the nation's workforce of the future.” This evaluation then, is undertaken from the perspective of the content developer stakeholder. As an instructional designer tasked with formulating content that follows the guiding principles of the ADL Initiative, I am evaluating this e-learning effort based on my own experiences along Rosenberg’s strategic foundation for e-learning.

Evaluation of the DoD ADL Initiative

Building an Infrastructure

The World Wide Web (WWW) has been used for some time to deliver training (Rosenberg, 2001). As technology has advanced, providing new opportunities for multimedia and interactivity, e-learning as a training option has become more and more attractive. Perhaps one of the key strengths of the Web as a training medium is its simplicity for users. Hypertext Transfer Protocol (HTTP) is the standard Internet protocol that allows the exchange of information over the web. Hypertext Markup Language (HTML) is a similarly common, non-proprietary language used to publish web pages. Together, these standards form a foundation for the widely accessible communications structure that is the WWW. Web designers can craft web-based content using nearly any platform—Microsoft® Word, Macromedia® Dreamweaver, Notepad, FrontPage—and the common, underlying standards that support the web will enable the content to be displayed the same for every user, regardless of the browser or computer on which it is viewed. The user has no need to know where the document s/he is viewing is located or how it is accessed; the process is transparent to the user (Netlingo, 2002).

While the Web has quickly become a universal delivery platform for much training, it has limitations. Essentially communication between web sites and the learner is one-way. The learner generally views information presented from a remote server without sending information back. From this inherent limitation was borne the Learning Management System (LMS). An LMS is “a suite of functionalities designed to deliver, track and report on and manage learning content, student progress and student interactions” (SCORM, 2002, p. 1-30). Building on the ever-improving communications and delivery platform of the Internet, LMSs take over where the Web ceases to be as useful. While there are literally dozens of LMSs on the market with a variety of functionality, all LMSs enable the creation of an environment in which the learner can control his/her own learning. This is a critical part of the ADL anytime, anywhere and just-in-time vision.

E-learning by its very nature is a technological solution, requiring a supportive technological foundation. Rosenberg (2001) supports the selection of a single LMS for distribution of e-learning content. The functionality added by these systems, particularly when facilitating an environment in which employees can access learning on their own, make the use of a common LMS essential. In an organization the size and breadth of the DoD, selection of a single LMS to meet the needs of all of its branches and divisions has not been managed. This has presented a bit of a problem, predicted by Rosenberg—a problem of interoperability (Rosenberg, 2001).

Today’s LMSs, while improving the ability to manage online courses, have created new obstacles. The LMS market is highly competitive. Many LMS developers have been swallowed by larger companies or simply vanished, leaving buyers of their systems with an unsupported product. LMSs are also proprietary and typically only support courses developed by the same company—they are not interoperable. That means that if an LMS company goes out of business, or if an organization wishes to buy a Commercial Off-The-Shelf (COTS) or custom product created by a competing vendor, the likelihood is that product will not be usable with the LMS in which that organization has invested. Simply, “current Learning Management Systems are not designed to satisfy the potentially 11.5 billion $US e-learning opportunities available by 2003. As a result vendors are losing money and clients of LMS systems are struggling with costs” (Thornborough, 2001).

The DoD is no exception. An example best illustrates the impact of this situation. The US Navy, more than any other military branch, looks to e-learning to provide training to its personnel. Navy E-learning, the online portal developed to serve Navy and Marine Corps personnel, civilians and dependents is powered by THINQ TrainServer LMS. This system launches, tracks and manages all courses for each individual user (Harris, 2002). But the DoD is large and fractured, and experience with LMSs is new; the decision to use the THINQ LMS is not pervasive. On a much smaller scale, a single squadron aboard Cherry Point, NC, is preparing for use of a different LMS—Air Combat Online (ACOL)—to launch, track and manage their courses. Courseware procured for that squadron will be designed and tested specifically to run on the ACOL LMS. In the current situation, without a common standard, any courseware designed to run on the ACOL LMS will not be usable with the THINQ LMS. If the Navy wishes to offer courses such as these via the Navy E-learning portal, it will be unable to do so without some sort of common standard that would enable courses designed by different vendors to run on different LMSs. Such a limitation can paralyze an e-learning effort (Rosenberg, 2001).

This presenting problem of interoperability is one of the key impetuses driving the development of the SCORM. “Very shortly, organizations utilizing LMS type systems will have the prerogative to move from any LMS vendor to any other LMS vendor and carry the investment made in courseware with them assuming of course that they license a SCORM-[conformant] LMS application” (Thornborough, 2001).

The SCORM and interoperability.

The Sharable Content Object Reference Model (SCORM) is a set of guidelines and specifications pulled together from various sources for the development of e-learning content, technologies and services. At incept of the ADL Initiative, many other organizations had already begun work on various aspects of e-learning technology and infrastructure. ADL sought to unite those efforts into a single, comprehensive reference model that could be used to test the effectiveness and realistic application of the standards and specifications it brought together. In particular, the work of four key players, the Alliance of Remote Instructional Authoring and Distribution Networks for Europe (ARIADNE), the Aviation Industry CBT (Computer Based Training) Committee (AICC), the Institute for Electrical and Electronics Engineers (IEEE) Learning Technology Standards Committee (LTSC), and the IMS Global Learning Consortium is brought together to form the SCORM. (SCORM, 2001).

As a reference model, the SCORM is itself not a standard or a specification to which one may comply. Rather, it is a collection of standards and specifications written by organizations such as the IEEE, AICC and IMS. The SCORM works with these standards bodies to integrate these standards and specifications into a single, logical and useable model. It then serves to test them and apply them in real-life ways to assess their feasibility. LMSs and courseware that conforms to the standards and specifications contained in the SCORM are referred to as being SCORM-conformant.

In January 2000, Version 1.0 of the SCORM was released. On initial release, the model was called the Sharable Courseware Object Reference Model. The first version of SCORM sought to enable:

LMSs to launch content authored by different tools and communicate with that content
Different LMSs to launch the same content the same way and communicate with that content
LMSs to access a common repository of learning content and to launch that content
Entire courses to be moved from one LMS to another with the same results.

Subsequent versions of the SCORM changed the name to the Sharable Content Object Reference Model to acknowledge that the specifications can apply to various levels of courseware components (lessons, modules, units, segments) rather than just courses. This difference illustrates the movement of the ADL Initiative toward reusable courseware and content. These courseware components or packages of learning content, called Sharable Content Objects, or SCOs, can be plugged into any SCORM-conformant LMS and “played” as intended without the conflicts that could typically arise with incompatible and proprietary LMSs. The current version is SCORM 1.2 with release of 1.3 pending.

The SCORM is based on five high level requirements guiding the ADL Initiative: accessibility, interoperability, durability, reusability, and cost effectiveness. For each of these guiding objectives, the SCORM incorporates features to support their achievement. The SCORM’s goal of facilitating interoperability specifically addresses Rosenberg’s concerns about a single LMS and the movement of e-learning content throughout an organization seamlessly, and without conflict. Interoperability in the SCORM refers to the ability of a SCO to be served on any SCORM-conformant LMS, regardless of the tools used to build the SCO. A technical standard in the SCORM specifies how an LMS and a SCO should communicate, including launching and finishing the SCO, and exchanging tracking and progress information of the learner in the material. All of this communication is transparent to the learner, but critical to its success is the specific content and packaging standards provided by the SCORM that support interoperability. These technical standards ensure that all content developers can create learning content that will be “playable” on any LMS within the DoD.

Accessibility and a single portal.

With the establishment of a single or interoperable LMS system, Rosenberg (2001) highlights two other criteria for implementing a successful infrastructure for e-learning. Ensuring that users have access to e-learning content is a fundamental need. Accessibility can actually have two interpretations in this context, however. Rosenberg means it to imply that users must have adequate connectivity via a basic, reliable Internet connection. From the perspective of a contracted content developer, however, providing such connectivity is not the domain of the DoD. Facilitating access to learning content by content developers once connectivity is established, however, is a critical part of the ADL Initiative. Again, the ADL Initiative turns to the SCORM for such a goal. Instructional content that is accessible can be located and accessed from any remote location and delivered to other locations. To address this requirement, the SCORM contains a content packaging specification that defines how SCOs should be packaged and labeled such that they can be located and accessed for use. Per this standard, learning content may be identified by its detailed label, called metadata, pulled from any remote location and served to the learner on a SCORM-conformant LMS.

Providing access to such DoD learning content through a single portal would allow learners and content developers to find e-learning and non e-learning solutions easily. This portal could be personalized to meet the needs of each learner or content developer such that each could specify links and content that are important to them. Multiple, fractured portals throughout an organization increase the likelihood of redundancy, internal competition and chaos due to lack of coordination (Rosenberg, 2001). It would seem from the SCORM documentation that such a goal is part of the vision of the ADL, indicated by the reference to the development of a common repository of learning content for access by an LMS. The development of such a repository and portal, however, has not yet taken place. If content will eventually reside on many distributed repositories however, it will be in the best interest of the DoD and its constituents, given the points emphasized by Rosenberg, to develop a single portal to access all repositories.

Building a sound technological infrastructure for an e-learning initiative is a critical first step to any effort. Rosenberg (2001) highlights several key criteria to ensure infrastructure success. The DoD has begun to address the many technological challenges of any e-learning initiative through the creation of the SCORM, a reference model designed to facilitate interoperability and accessibility, among other goals. Table 1 lists Rosenberg’s key criteria, briefly describes what these criteria are, and summarizes the action taken by the ADL Initiative to address the key criteria.

Rosenberg’s Criteria	Criteria Explained	ADL Initiative Action
Selection of a common LMS / has a goal of interoperability.	The organization selects a single LMS to ensure portability of learning content.	SCORM standards ensure interoperability of learning content across LMSs.
Learning content is accessible.	Users have adequate connectivity and can access learning content.	Connectivity of content developers is beyond the scope of DoD. SCORM standards ensure content accessibility through metadata and a content packaging specification.
Single portal entry.	The organization builds a single portal for access and retrieval of e-learning content.	SCORM documentation seems to speak to a single content repository—but it is as yet undefined.

Table 1

Building on Infrastructure: Combining Online Training and Knowledge Management

Rosenberg (2001) strongly advocates the integration of Knowledge Management with online training in any e-learning strategy. His argument for such a marriage stems from his vision of the role of today’s training function. Looking beyond the traditional paradigm of training for training’s sake, Rosenberg embraces a training mission that supports all of the ways in which people learn and that positions itself at the business table with other key organizational partners. By taking a Knowledge Management approach to an e-learning initiative, the organization acknowledges the value of information and instruction to learning and moves beyond the notion that only training has value in improving the performance of workers in an organization (Davenport & Prusak, 1998, Rosenberg, 2001, Stewart, 1997, Tobin, 1998).

To successfully implement a KM solution as a part of an e-learning initiative, Rosenberg (2001) advocates for several key developments: demonstrations of success, development of an oversight group, support for collaboration, incentive for participation, and the development of a knowledge structure.

Demonstrating success.

By demonstrating success and progress along the way, Rosenberg (2001) suggests that an organization can build support for an e-learning initiative. The ADL group has taken this approach very seriously. The collaboration of participating organizations like the IEEE and IMS Global Consortium with ADL continues as the SCORM evolves. Twice annually since the SCORM’s birth, the ADL Co-Laboratories have sponsored “Plugfests” to provide such a forum for representatives of government, industry and academia interested in the SCORM to interact in an attempt to synchronize the development of Learning Management Systems, training content, and authoring tools with the SCORM. These Plugfests allow vendors to test their products against the reference model and dialogue on the progress, strengths and weaknesses of the SCORM. To date, 7 plugfests have been hosted by the ADL Co-Labs with increased participation and interest in each subsequent event. The most recent event was attended by representatives from more than 100 private companies and organizations that provide training services to the DoD (Book, 2001). Once these standards and specifications that make up the SCORM have been tested and improved or rejected, the goal is the emergence of an acceptable, usable standard for e-learning (ADL, 2002).

Oversight group.

Recognizing that development of a comprehensive e-learning strategy that incorporates a KM component is a big job, Rosenberg (2001) asserts that management and people are a core component. He recommends the development of an oversight group, “composed of all the KM stakeholders, who set policy, guidelines, standards, and practices for the system, and a core team of specialists that implement them on a daily basis” (p. 107). It would seem that the ADL Initiative is not quite ready for this assignment of duties. Without the existence of a content repository, there is no true system yet for which to formulate guidelines, standards, and practices. Before such a system is implemented, however, the DoD should create such a group of stakeholders to ensure the effort is adequately managed. It’s not too early to be thinking about it now.

Collaboration support.

Rosenberg (2001) encourages organizations to move beyond the simple storage and retrieval of documents and incorporate an opportunity for users to collaborate with each other. Through the ADL web site (2003), the ADL Initiative provides an outlet for content developers to interact with one another, ask questions and express concerns through forums dedicated to implementing the SCORM and preserving instructional integrity. But this is a limited approach to providing collaboration.

In the vision of the ADL Initiative, learning objects will be accumulated, tagged, and stored in repositories for broad distribution and use. Learning objects will be searchable based on metadata tagging for easy retrieval. Users will also be able to submit new learning objects to the repositories for storage and retrieval. This strategy represents level 2 Knowledge Management according to Rosenberg (2001)—information creating, sharing and management. In this type of system, information is continually updated with users contributing to the growth of the knowledge database.

Rosenberg (2001), in describing a knowledge management effort, is referring to having a system in place that supports users in accessing both instruction and information. His ideas have a great deal of application in the world of developing and reusing learning objects. “Knowledge management fosters the growth of knowledge communities, learning communities, or communities of practice...” (p. 79). The value of these communities is huge. By allowing members to interact and share ideas and experiences, it is possible to reduce much confusion and isolation associated with a new way of doing things (Stout, Slosser & Hays, 2001).

Recker and Wiley (2000) suggest that it may be possible to create such a community of users of learning objects by taking advantage of the flexibility of metadata. The authors posit that users of learning objects have much to share with one another regarding the effective use and reuse of learning objects. In non-authoritative metadata, information about an object that is not fixed and persistent, a content creator has the opportunity to provide the contextual and changing aspects of a learning object, such as the context and surrounding activities in which is was used. Further, non-authoritative metadata could describe the community of users from which a learning object is derived. Using what Recker and Wiley call a collaborative information filtering approach, users of an object may offer recommendations about the qualities and usefulness of a particular learning object. Anyone may contribute to an object’s metadata in this approach, but information regarding each contributor would be recorded, thus facilitating the growth of a connectable community of users. “While much learning object work has been set in a direct instruction paradigm, this kind of community building through learning object rating and use provides a model for learning object reuse within a more open-ended, constructivist learning environment” (Recker & Wiley, 2000). Through collaboration like this, the integration of KM effort into the overall ADL Initiative could result in a stronger, more connected community of users.

Rewarding participation.

Incenting users to contribute to the system and providing rewards for participation is critical to the success of a KM system and the e-learning strategy as a whole ((Davenport & Prusak, 1998, Rosenberg, 2001, Stewart, 1997, Tobin, 1998). In order to support the ADL Initiative, learning content developers must be motivated to change the way they design content, have the skills and knowledge to successfully do so, and have access to the resources to do so. Currently, to meet the criteria of SCORM conformance, content developers can package a lesson or a course as a SCO, fill in the requisite metadata fields, produce the appropriate SCORM documentation and voilá—their job is done. But I don’t believe this to be in the true spirit of the ADL Initiative. To achieve maximum reusability, SCOs ought to be smaller than a lesson or a course. But breaking learning content into smaller chunks isn’t easy. Thinking in terms of small, reusable pieces of information and instruction is a different approach. Completing metadata for more SCOs is tedious work. Therefore, more SCOs means more work. And the SCORM documentation itself says SCOs can be any level of granularity. Where is the incentive for a content developer to move beyond just being conformant to producing a better reusable product? The prototype study by Stout, Slosser and Hays of the work of ADL early adopters indicates there isn’t any incentive there yet. The researchers found that the lack of guidance on what constitutes a SCO left little incentive on the part of developers to “go lower.” Metadata tagging was referred to as “an arduous process” in need of better guidance to those tasked with the process. Specifically, those tasked with metadata tagging expressed a lack of confidence in the value of their tagging and whether the consistent application of an intuitive and meaningful scheme was occurring (2002). What’s needed here is vision and understanding.

In the beginning of the development of a learning object economy, the DoD is faced with empty coffers of learning object repositories to fill. That’s daunting work for the vendors and developers of learning content. With so much doubt and confusing swirling around the ill-defined nature of learning objects and questions about granularity, combination and reuse, there is little confidence regarding the usefulness of any effort put forth at this time in designing objects for reuse. In their prototype study, Stout, Slosser, Hays found that

“more universally expressed than any other issue was the frustration on the part of interviewees with what they perceived to be a standard, which did not allow them to build instructionally sound content. Furthermore, the frustration extends to feelings that anything currently done according to the “standard” will need to be undone/redone in the future. The SCORM user community is crying out for more guidance regarding SCORM, as well as more realistic milestones of what can and cannot be expected in future versions of it” (2001, p. 4).

Simply, if content developers do not see reusability as a viable part of their future jobs, they won’t be motivated to create truly reusable content. The onus is on the ADL team to provide better instruction, clearer definition and stronger guidelines on the development of reusable content. This can only take place with the refinement of the issues not addressed in current versions of the SCORM. In the meantime, the ADL Initiative will be hard pressed to expect more from content developers.

Knowledge structure.

Finally, a key criteria emphasized by Rosenberg (2001) in building an e-learning strategy that incorporates a KM solution is to develop a knowledge structure. This structure should create logical links between content elements that allow them to fit together seamlessly and logically. Content, whether information or instruction, should be structured so that users can access just what they need when they need it. For content developers designing and developing content for the DoD to fit into such a system, this is a critical weakness of the ADL Initiative.

The ADL Initiative emphasizes anytime, anywhere delivery of instruction tailored to the individual. It is content or learning objects that make this possible. In the vision of ADL, learning objects will be stored in content repositories, ready to be called on for the real time assembly of instruction as needed by the learner. Once again, the SCORM assumes a role of primary importance here. It is the SCORM that provides the technical specifications such that learning objects can be easily and transparently shared across learning delivery environments and seamlessly and logically linked together to meet the needs of learners (SCORM, 2001).

Derived from the object-oriented paradigm of computer science, the idea is to break learning content into small, discrete “chunks” that may stand alone, or may be reused in multiple contexts (Shook, Dargue & Carlton, ND). While the concept of learning objects is generally understood in the training community, the exact definition of what a learning object is varies greatly. Some of the variations include:

“A learning object is a self-standing, discrete piece of instructional content that meets a learning objective” (Masie Center, 2002).
“A learning object, for all practical purposes, is an object or set of resources that can be used for facilitating intended learning outcomes, and can be extracted and reused in other learning environments (Mills, 2002).
“A learning object is a self-describing, self-contained small chunk of learning that accomplishes a specific learning objective” (Oakes & Rengarajan, 2002).
“…a learning object may be one of any number of items: a map, a web page, an interactive application, an online video – any element that might be contained inside a course” (Downes, 2001).
Learning objects are “…independent chunks of educational content that provide an educational experience for some pedagogical purpose” (Quinn & Hobbs, 2000).
“A learning object is defined as any entity, digital or non-digital, that may be used for learning, education or training” (IEEE LTSC, 2001).
A learning object is “any digital resource that can be reused to facilitate learning” (Wiley, 2000).
“You can think of learning objects as knowledge granules created by specialists throughout an organization and that are accessible to many others in the organization” (Clark, 1998).

How learning objects are defined is a key missing piece of the SCORM and is open to a great deal of criticism and speculation. With no common and understood or agree-upon definition, there can be no uniformity in content creation. It is here that Rosenberg’s emphasis on new approaches to e-learning to include both instruction and information is applicable. Creation of learning objects for use and reuse needs to be based on a sound knowledge of how they will support an effective e-learning effort. According to Rosenberg, to support learning, learners need access to both training and information (2001). Information simply presents facts to the learner to be assimilated and turned into knowledge. Instruction, on the other hand, “involves directing students to appropriate learning activities; guiding students to appropriate knowledge, helping students rehearse, encode, and process information; monitoring student performance; and providing feedback as to the appropriateness of the student’s learning activities and practice performance” (Merrill, et. al., 1996). It follows that how learning objects are defined in the SCORM should have something to do with whether those objects may be used for instructional or informational purposes. Clearly, there is room for both. If both are to exist, however, they must be supported.

Common to nearly all definitions of learning objects are two conditions: that learning objects stand alone and that they can be used in multiple places. There is decidedly less consensus, however, about the granularity and combination of learning objects. Granularity refers to the size of the object, and combination refers to how the objects are assembled into a meaningful whole (Wiley, Gibbons & Recker, 2000).

The SCORM does not define learning objects directly, but instead applies a taxonomy to learning objects grouping them into categories of varying size. The SCORM defines 3 levels of learning objects: assets, SCOs, and content aggregation. An asset is a single media object, such as a sound file or graphic, a SCO is a collection of one or more assets, and an aggregation is a collection of SCOs. In this taxonomy, the granularity of the learning objects is a function of the number of smaller media elements that have been combined to form the larger object (Wiley, Gibbons & Recker, 2000). If the learning object is planned to be informational in purpose, this approach seems reasonable. If we consider how learning objects will be reused and combined to form an effective piece of instruction, however, there is room for concern.

Learning objects as instruction.

While theories of learning abound, there is general agreement in the world of instructional design that specific instructional events presented in a specific order maximize the effectiveness of the instruction and likelihood that learning will occur. These events include: introducing and connecting the learner to the material, presenting or demonstrating the material, guiding the learner in interacting with the material and providing feedback, and allowing the learner to practice the material (Gagne, Briggs & Wager, 1992, Rothwell & Kazanas, 1998, Silberman, 1998, Cantor, 1992, Bowman, 1998, Gagne & Medsker, 1996). If a learning object is defined in terms of the number of media elements it contains, what does that tell us about its instructional purpose and use?

The level of granularity of a SCO in terms of combination of media elements is a gray area in the SCORM (2001), and deliberately so:

SCOs are intended to be subjectively small units, such that potential reuse across multiple learning objectives is feasible. The SCORM does not impose any particular constraints on the exact size of a SCO. During content design and authoring activities, when determining the size of a SCO, thought should be given to the smallest logical size of content that one might desire to have tracked by a LMS at run-time. It is intended that the content developer will determine the size of the SCO based on how much information is needed to achieve the learning outcome and on the level of reuse that the content developer wishes to obtain (pp. 2-4 – 2-5).

But if anything is a SCO and, by association, a learning object, from a single graphic or web page to an entire course, how can LOs be reused and effectively and meaningfully assembled so that instruction may occur? How does grouping varying levels of media elements ensure that effective instruction will occur through the events of instruction? The answer is sobering: it doesn’t.

This has been demonstrated by early adopters of the SCORM, studied during a prototype ADL effort sponsored by the Joint ADL Co-Laboratory. These content developers found that varying decisions on the granularity of a SCO had important implications regarding the combination and effective reuse of a learning object. For example, if some content developers chunked SCOs at the course level, while others chunked SCOs at the individual screen level, developers found that it is nearly impossible to reuse and aggregate these objects into a coherent piece of instruction (Stout, Slosser & Hays, 2001).

The ADL Initiative specifically describes the SCORM as being “pedagogically neutral” (ADL, 2002, p. 8) with instructional design issues beyond the scope the ADL mission. When learning objects are to be used as instruction, this is a dangerous approach to take. Rosenberg (2001) lists a focus on form over substance as one of the key reasons much computer-based training fails. Agrees David Merrill, a well-known expert in instructional design from Utah State University, “The problem in instructional technology today is that we are all busy doing things, not studying things. Nobody is studying the effectiveness of what we’re doing. That’s how you create irrelevant fads” (Zemke, 1998, p. 3). Dr. Merrill asserts that instructional standards must be developed in addition to technical standards in order to ensure that the instructional learning objects supported by the SCORM are instructionally sound (Welsch, 2002). He has a valid point. It doesn’t matter how many LMSs on which an instructional learning object will “play” successfully if it doesn’t effectively teach or is of no use to the learner.

Dr. David Wiley, editor of The Instructional Use of Learning Objects and a thought leader in learning objects and instructional design, asserts that in order for instructional learning objects to be used effectively, a taxonomy must be constructed that is compatible with multiple instructional design theories. Toward that end, Wiley identifies five different types of instructional learning objects: fundamental, combined-closed, combined-open, generative-presentation, and generative-instructional.

Fundamental learning objects resemble SCORM assets and the most fundamental unit of a SCO. These are the most basic form of learning content and represent a digital resource that is not combined with any other. The purpose of this type of object is generally to illustrate or provide an example of a simple function (Wiley, 2001). A digital photograph or rendering of the Caution / Advisory panel in an AV-8B cockpit is an example of a fundamental learning object.

A combined-closed learning object is made up of several digital resources. The “closed” nature of the object means that the individual digital resources may not be pulled and reused from the object. This group of digital resources is combined at design time for the purpose of providing instruction or practice (Wiley, 2001). An example of a combined-closed learning object is an animation of a pilot’s hand selecting the cockpit controls for aircraft start-up. Included in the animation might be an audio depiction of what sounds are heard during start-up. Presented as a compressed SWF file created in Macromedia Flash, the object may not be separated into its constituent parts of still images and sound for individual reuse of those elements.

Combined-open objects, like combined-closed ones, are created from the combination of many digital resources, such as images, sound, text and animation. This combination is performed by a computer in real time. Unlike combined-closed objects, however, the individual elements that make up the combined-open object can be extracted and reused. These objects may include content presentation and / or practice. A web page that assembles fundamental and combined-closed elements together is a combined-open object (Wiley, 2001). For example, a web page that displays a digital photograph of a Multipurpose Color Display (MPCD) in the cockpit, launches an animation of the selection of a pushbutton on the MPCD when the user clicks on the image, and provides instructional text describing the action is an example of a combined-open object. The computer assembles the components into a web page in real time and each of the elements on the page may be extracted and reused.

Generative-presentation objects can be used to present reference material, instruction, practice or assessment. These objects use logic and structure to combine or generate and combine lower-level objects, including fundamental and combined-closed objects for these purposes (Wiley, 2001). An example is a JAVA applet that displays multimeter readings and caution light displays to test a learner’s ability to identify faults in an aircraft system.

Generative-instructional objects combine fundamental, combined-closed and generative-presentation object using logic and structure to instruct, practice and evaluate learner interactions with these combinations (Wiley, 2001). An interactive troubleshooting lab that demonstrates discrepancy identification and troubleshooting procedures, allows the learner to perform the procedures, and evaluates the learner’s ability to perform the task successfully is an example of a generative-instructional object.

Wiley (2000) also presents a set of prescriptive guidelines for selecting an instructional learning object type that provides “a bridge for the designer to follow from instructional design to learning object design” (p. 82). When comparing Wiley’s taxonomy to the SCORM taxonomy, multiple levels of SCOs becomes apparent. Clearly, Wiley’s fundamental resource equates to a SCORM asset. The other four levels, however, marry up to SCOs with various instructional purposes. A content aggregation still effectively groups instructionally compatible SCOs into a cohesive learning experience.

Wiley intends that identification of a learning object’s type according to this taxonomy and its appropriate use and reuse would be accomplished using the object’s metadata. Under the current metadata system, there are no fields dedicated to providing information about the instructional utilization of the SCO. In other words, potential users of a SCO must examine the object itself to identify its potential for use in a specific instructional capacity, thus defeating the purpose of metadata. Instead, Wiley suggests differentiating between authoritative and non-authoritative metadata. Authoritative metadata is information about an object that is fixed and persistent, such a title, date created, and author. Non-authoritative metadata would include information about the usefulness of the object in a particular context (Wiley & Recker, 2000). Examples of such metadata fields might include Educational Instructional Architecture (Wiley, 2000) and subjective fields such as Resource Quality or Educational Relevance (Wiley & Recker, 2000).

Wiley’s taxonomy and guidelines represent but a single effort by the ISD community toward linking ISD standards to the technical standards currently in the SCORM. In their article Learning Objects and Instruction Components, Quinn and Hobbs (2000) also site the absence of instructional standards in current learning object discussion and suggest the development of learning objects per their role in the instructional process. For example, instructional learning objects would be characterized as an introduction, a concept, an example, practice or a reflection and this information would be reflected in a specific metadata field. In earlier works, Wiley made the same observation. Prior to refining and developing his Learning Object Design and Sequencing (LODAS) theory with the taxonomy described above, he, too, advocated development of learning objects along the critical events of instruction (Wiley, 2000).

In practice, other efforts have been made to create a useful taxonomy of instructional learning objects to describe granularity and combination based on instructional design principles. Cisco Systems’ RIO, or Reusable Information Object, represents the networking giant’s efforts in determining a learning object approach. In this system, a RIO contains learning content, practice and assessment components. RIOs can then be grouped together to form a Reusable Learning Object, or RLO. The RLO includes the content of each RIO, plus an introduction, summary, and comprehensive assessment. Thus, an RLO represents all of the components needed to meet a learning objective (Barron, 2000).

There is a dearth of research regarding the instructional design implications of learning objects. Wiley and a few others act as pioneers in drawing attention to the need and filling the gap. But much more is needed. And a standard or consensus method must be found in order for such a taxonomy to be useful. “...The introduction of [metadata] fields conveying instructional design information...combined with a redefinition of granularity...could facilitate an immediately technologically implementable method of delivering individualized, or mass-customized, instruction” (Wiley, 2000, p. 10).

Learning objects as information.

In the SCORM v. 1.2 documentation (2001), there is evidence that the ADL Initiative understands that learning is not training. The mission includes a reference to education, training and “decision aiding (‘mentoring’) materials” (p. 1-11), which would indicate an appreciation for the value of both information and instruction. Conversely, the document limits the scope of learning objects by referring to the creation of “reusable learning content as ‘instructional objects’” (p. 1-11) and the development of “an instructional object economy” (p. 1-12). This view of learning objects as pieces of instruction to be combined to deliver training anytime, anywhere critically limits the effectiveness of the e-learning initiative by failing to recognize the value of information.

Rosenberg (2001) highlights performance support as a classic example of the usefulness of information. Performance support refers to job aids such as checklists or reference cards or other tools that enable productivity improvements on the job. Technology has already been tapped to provide this type of performance support. Electronic Performance Support Systems (EPSS) enable workers to be more productive with less effort.

Consider our AV-8B avionicsman from the opening scenario of this paper. Aircraft maintenance technicians do not work from memory. Rather, they are trained specifically to work from technical publications, particularly in the performance of troubleshooting and repair. With new equipment like the LITENING II Targeting Pod appearing regularly, keeping technical publications updated is a challenge. Developers and manufacturers of these systems quickly found that publishing these manuals electronically make them easier to use and update. These electronic versions have been around as long as 15 years and are called Interactive Electronic Technical Manuals (IETM).

IETMs do not instruct learners on a concept, but instead provide procedural directions to a user (Shook, Dargue & Carlton, 2002). That does not diminish their usefulness—IETMs still support learning and performance. “It has been demonstrated that the use of such Interactive Electronic Technical Manuals…can increase performance levels of inexperienced technicians to that of experienced technicians in complex tasks (e.g. fault isolation) and, correspondingly, can reduce the time required to provide a student with a given level of technical competence” (Jorgensen, Fuller, Post & Rainey, 1995, p. 7).

IETMs are developed in much the same way that training is developed. In depth front-end analyses are performed, producing technical documentation on the use and repair of new equipment. This documentation is often the source of content for training developed for new products. By integrating IETMs and training, the Navy has found that “the IETM can be profitably coupled with other training systems, such as automated courseware preparation, course management, telecommunications, information management, and database systems, in a way that permits significant increases in the effectiveness of all phases of training…” (Jorgensen, Fuller, Post & Rainey, 1995, p. 13).

Boeing, a major supplier of products to the DoD, has applied concepts of the SCORM to the design of IETMs and online training with promising results. Because informational content contained in technical publications becomes the source of content for training materials, Boeing applied the concepts of informational and instructional learning objects. The group defined SCOs as informational objects for use in the IETM. The same SCOs were then supplemented with instructional characteristics, such as an introduction, objective, practice and assessment for use in online training. In this way, informational SCOs were reused to form instructional SCOs (called “learning objects” by the Boeing team) and content was only developed once, achieving one of the high level requirements of the ADL Initiative (Shook, Dargue & Carlton, 2002).

Much ado has been made about the instructional use of learning objects, and rightfully so. If online training is to succeed using learning objects, clearly some attention to instructional design must be given. Less thought, though, seems to be focused on the informational use of learning objects. Wiley (2000), in arguing for ISD standards asks, “where is the learning in learning object?” We know that learning can take place without well-designed instruction. So in every case, perhaps a clear instructional purpose for a learning object is unnecessary. Boeing has effectively demonstrated that learning objects with informational use are equally valuable. What is needed is a distinguishing marker for informational vice instructional objects.

In her article Recycling Knowledge with Learning Objects, Ruth Colvin Clark (1998) argues for just such a taxonomy. Information objects, according to Clark, would be used to deliver facts, concepts, processes, procedures and principles to learners. The type of information object and its domain (such as AV-8B avionics system maintenance) could be indicated by a metadata tag. Instructional objects, on the other hand, would be much the same as Wiley and others’ learning objects. These objects would include the content provided in information objects, and in addition, learning objectives, practice, feedback and assessment. Metadata tagging to characterize the instructional use of the object as described by the ISD evangelists would apply.

Currently, the SCORM, with its SCO taxonomy, can accommodate both of these perspectives. Because of the flexibility inherent in SCO granularity and combination, learning objects may be instructional or informational and still adhere to the definitions as set forth in the SCORM. What is required, however, is more attention to effective metadata tagging to incorporate these descriptions. In order for reuse to become a reality, users must have a clear idea of the intention and nature and potential use of a learning object.

Merging a KM system with online training within an e-learning strategy greatly enhances the effectiveness of the system (Rosenberg, 2001). Rosenberg (2001) highlights several key criteria to ensure success of such an implementation. The ADL Initiative, focused primarily on delivery of instruction, tends to fall short on many of the techniques Rosenberg and others suggest in the expansion of the organizational vision to include learning solutions other than training. Table 2 lists Rosenberg’s key criteria, briefly describes what these criteria are, and summarizes the action taken by the ADL Initiative to address the key criteria.

Rosenberg’s Criteria	Criteria Explained	ADL Initiative Action
Generating support through demonstration.	The organization shows how the system works.	Plugfests allow participants to test their courseware and LMS for SCORM conformance for inclusion in a DoD content repository.
Developing an oversight group.	The organization provides management of content and establishment of common policy, guidelines and standards enhance the usability of the system.	None known at this time beyond establishment of the 3 Co-laboratories.
Building in collaboration.	The organization provides an opportunity for users to collaborate with one another.	ADL web site has forums for asynchronous communication.
Providing incentive and reward for participation.	The organization incents users to share information and content and follow the guidelines and policies established.	None other than contract mandates for “SCORM-conformance.”
Developing a knowledge structure.	There are clear logical links and tags between content elements so that content can seamlessly and naturally be assembled as needed.	Metadata tagging is available, but poorly defined SCOs make combination and sequencing a potential mine field for developers.

Table 2

Building a Learning Architecture

E-learning is not a panacea. In his book, Rosenberg (2001) argues diligently for wise and selective use of electronically delivered instruction and information. Instead, he champions a combined approach through the development of a learning architecture that integrates both e-learning and traditional classroom training programs. It is through this type of design, he asserts, that e-learning is most effectively employed in the way it is intended.

The ADL Initiative assumes a web-based delivery approach and is focused on the leading edge of technological advances. Dan Rehak, one of the chief architects of the SCORM, recognizes the limitations of this approach and argues that the ADL is not designed to replace all other instructional approaches. He asserts that the initiative and the underlying SCORM infrastructure “is essentially about a single-learner, self-paced and self-directed. It has a limited pedagogical model unsuited for some environments,” (Kraan & Wilson, 2002, p. 1).

For some in the DoD, this is an important concession. The military relies heavily on instructor-based education for certain types of learning and performance outcomes. Forcing SCORM-conformance on some current online courses and “hybrid” courses that combine e-learning and classroom training may render them obsolete, costing the DoD thousands of dollars in development costs. In his article Much Ado about SCORM, author Ethan Smith (2002) profiles a particular hybrid course, Acquisition 201 / Intermediate Systems Acquisition Course (ISAC) created by the Defense Acquisition University (DAU), one of the military’s premier training institutions. The course integrates approximately 60 hours of self-paced, online courseware with classroom training to cover more than 150 learning and performance objectives. Developed four and half years ago, the course is not SCORM-conformant. The course utilizes a scenario-based instructional approach, with each lesson building on the previous lessons as a story unfolds. Because the material in the course is highly contextualized, it would require a complete redevelopment to create the stand-alone learning objects required by the SCORM.

The importance of instructor-led instructional components must not be lost as the ADL Initiative moves forward. The Defense Language Institute Foreign Language Center (DLIFLC) estimates that 30 percent of each of its advanced language courses is instructor-led, most often by a native speaker. For personnel at the center, this method of delivery most suitably meets the needs of the topic and students by providing human interaction (Smith, 2002). The ADL Initiative seems to recognize that this is often the case with its admission that it targets training for specific purposes (Kraan & Wilson, 2002). The danger, however, is that SCORM and the ADL Initiative “might be seen as the end-all, cure-all approach to training in the interest of usability, to the detriment of the student’s learning and long-term retention” (Smith, 2002, p. 35).

To avoid the global and sweeping selection of e-learning as a solution in every learning situation in response to the promotion of the SCORM, the DoD would be wise to employ some simple guidelines to make educated decisions as to which designs are most effective in which situation for learning and retention. In his book, Rosenberg (2001) provides some useful guidelines.

By conducting a thorough needs assessment, analyzing the target audiences and understanding the performance gap, a more accurate and appropriate media selection could be made. In many cases, e-learning might well be the best solution. In others, however, classroom instruction, or a combination of both may be most suitable. Generally, web-based solutions are best suited for cognitive skills, while instructor-led environments are more useful for teaching psychomotor and attitudinal ones (Driscoll, 2002). Blended solutions combining web-based instruction with traditional classroom teaching are more and more popular, as they reduce costs by moving some classroom learning to the Web, encompass the best of both worlds for the three learning domains, and make use of the benefits of technology without becoming prisoner to its limitations (Driscoll, 2002, Beer, 2000). In my experience, however, many contracts with the DoD do not allow much decision room in media analyses. While a front-end or training situation analysis is conducted as part of many projects, often the media—in most cases, web-based media—is pre-selected for the project. Often it is the job of the contractor to best make the learning solution match the media as prescribed. Obviously, this is a less than ideal situation.

Other criteria suggested by Rosenberg (2001), including basing learning design on key job competencies, understanding the business need and testing learning architecture with key stakeholders all are influenced within the DoD by this prescribed selection of media, often before a contract is even bid out. It has been my experience that, while working with the customer to best craft a positive and effective learning experience is a mutual goal, we often do that collaboration within the parameters set forth by the contract. Rapid prototyping helps create a learning design that best satisfies the customer and most effectively delivers the instruction, but independent selection of media based on these other criteria just doesn’t happen in my experience.

In the design of any learning experience, Rosenberg (2001) recommends that learning material be reused as often as possible. On this, the ADL Initiative has expended a lot of thought and effort. The creation of reusable learning content is one of the guiding criteria of the ADL Initiative. One of the five goals of the SCORM is to ensure reusability. Reusable learning content is flexible enough to be incorporated into multiple applications and contexts. All of the standards set forth in the SCORM, specifications for packaging and labeling content and for communications between content and an LMS, work to support the effective and feasible reuse of SCOs. When users can locate information, pull it and use it without rework on a SCORM-conformant LMS, they have effectively reused that learning content. SCORM makes this technically possible.

Rosenberg’s final criteria, using the Web as a unifying portal for blended solutions, developing a community on the web, and engaging learners in learning programs, all presume a learning architecture that incorporates both the web and the classroom. Unfortunately and as discussed, this is not a given situation in DoD contracts. The reality is that SCORM-conformance is fast becoming the primary filter through which the DoD views possible vendor solutions. Indeed, with the launch of the ADL certification program in 2002, vendors may now verify that their content and management systems conform with the SCORM v. 1.2 and no doubt improve their chances for winning contracts (ADL, 2003). But if an e-learning strategy is to succeed, it must not focus on a technological solution to the exclusion of all other things.

Providing learning solutions that maximize the effectiveness of web-based and traditional learning environments is important to any e-learning strategy (Beer, 2000, Broadbent, 2002, Driscoll, 2002, Rosenberg, 2001). Rosenberg (2001) highlights several key criteria to guide the appropriate selection of media for a learning situation. The ADL Initiative, while strong on the principle of reuse, is specifically targeted and in support of pure and global web-based learning solutions. Table 3 lists Rosenberg’s key criteria for design of a learning architecture, briefly describes what these criteria are, and summarizes the action taken by the ADL Initiative to address the key criteria.

Rosenberg’s Criteria	Criteria Explained	ADL Initiative Action
Conducting an analysis, basing design decisions on competencies, business needs and stakeholder input.	The organization makes decisions on learning design based on factors such as audience needs, business need and learning domain.	Currently, the ADL Initiative focuses on web-based solutions to the exclusion of all others. Many DoD contracts pre-suppose the web as the delivery media.
Reuse existing content.	The organization avoids reinvention of the wheel by reusing or repurposing existing learning material when possible.	The SCORM supports and facilitates the reuse of learning content with specific standards on reusability.
Use Web as unifying portal for learning solutions; create community online; engage learners.	The organization takes steps to create a learning environment on the web that supports the blended approach to learning.	Currently, the ADL Initiative focuses on web-based solutions to the exclusion of all others. Many DoD contracts pre-suppose the web as the delivery media.

Table 3

A Sound Business Case

The ADL Initiative, with its heavy emphasis on the development of technical standards, has recently come under fire for its perceived failure to demonstrate a connection to measurable learning outcomes (Greenagel, 2002). “[The SCORM] are not standards that treat learning outcomes, but instead deal with tagging, coding and indexing Learning Objects to facilitate reuse of digitized training materials. Some have likened that effort to ‘rearranging the deckchairs on the Titanic’…No one knows…whether [an] LO has ever resulted in anyone learning anything or subsequently demonstrating any competency” (p. 2). That such a criticism should arise from the training and development community is not a surprise. Evaluation as a field has often been overlooked in organizations, but has matured into a profession and an integral part of many organizational learning efforts (Broadbent, 2002, Russ-Eft & Preskill, 2001). With an increased emphasis on government accountability and a need for organizational leanness, efficiency and global competitiveness, it is to the benefit of any organization, including the DoD, to examine the business case for an e-learning initiative.

For many organizations, the justification for e-learning is a challenge. As Greenagel (2002) asserts in his impassioned plea for measuring learning outcome with an e-learning solution, Rosenberg (2001) agrees that evidence of performance improvement by learners is an important measure of e-learning effectiveness. He argues however, that increased learner performance is not sufficient proof. Evaluating business performance along four measures, cost, quality, service and speed forms a true foundation for e-learning justification (Rosenberg, 2001).

Cost.

A cost measure answers the question, “will the ADL Initiative save the Department of Defense money?” (Rosenberg, 2001). Studies analyzed by the ADL group have found that technology-based instruction reduces costs of achieving a wide range of instructional objectives by 30-60 percent. Additionally, the studies found a 30 percent reduction in time to achieve given instructional objectives or 30 percent increase of learner skill and knowledge depending on whether achievement or time were held constant (SCORM, 2001).

Each year, the United States military spends $17 billion to provide training and educational opportunities to its 2.5 million military personnel, Department of Defense (DoD) civilian workers, and their dependents (ADL, 2002). If the DoD could realize that 30 percent reduction in time to train by just 40 percent of all learners in specialized skill training, the DoD could save over $500 million annually (SCORM, 2001). These figures seem to have effectively convinced the DoD that the economics of an e-learning initiative are relentless.

Given the size and breadth of the DoD, the quantity of training activity that takes place within its ranks, and the number of parties involved in creating those training programs, including DoD customers and training vendors / contractors, it is inevitable that duplication of effort and expense will occur without some significant changes in the way training products are acquired. Consider the opening scenario of this report involving training design for T58 engine test cell operators. The curriculum for this training course included an introductory unit on basic electricity. As described, the likelihood is that somewhere within the layers of the USMC, the Department of the Navy, and the DoD, there already exists instruction on this topic. The ADL Initiative supports the development and use of reusable learning objects, seeking to eliminate this type of duplication of effort and realize cost savings in the process. For experienced trainers, the concept of reuse of instructional content is not a new idea, nor is it unique to e-learning solutions. Many trainers reuse their own content using such high-tech techniques as copy and paste. The SCORM standards though, seek to facilitate easy and efficient sharing and reuse of all types of learning content through the entire DoD and ultimately beyond, effectively opening access to learning content previously undiscoverable to many content developers (SCORM, 2001). The benefits of such a system go beyond the parameters of an e-learning solution.

Finally, the high-level requirement of durability guiding the ADL Initiative seeks to realize a cost savings to the DoD. Historically, much time and effort was required to convert existing courses and material for web-based application and for use on an LMS. This happened because learning content was not durable. That is, it did not withstand technology changes—it could not be used without having to be redesigned or reconfigured. The same standards and specifications in the SCORM that detail how content should be packaged and communicate with the LMS ensure durability. SCOs that are SCORM-conformant will play on any SCORM-conformant LMS without requiring recoding (SCORM, 2001). This saves programmer time and content developer time, transitively saving the DoD money.

Quality.

The quality of an e-learning initiative focuses on that measure of how well it improves performance and if that result was worth the cost (Rosenberg, 2001). The ADL Initiative seeks to provide just-in-time and just enough individualized instruction for each learner. According to research by Bloom cited in the ADL’s SCORM v. 1.2 (2001), “the achievement of individually tutored students may exceed that of classroom students by as much as two standard deviations...” (p. 1-18). The ADL Initiative then, hopes to duplicate the quality of individual tutoring through the type of individualized instruction made possible with e-learning.

As emphasized by Greenagel (2002), there is very little dialogue surrounding the ADL Initiative on assessing the effectiveness of e-learning. Rosenberg and others suggest evaluation along Kirkpatrick’s four levels: reaction, learning, behavior and impact. Kirkpatrick’s taxonomy provides a good starting point and framework for developing an evaluation philosophy and plan. The combination of learning objects into a training program, however, present some new and unique issues not found in traditional training situations that must be considered and addressed before a useful and effective evaluation can take place.

To plan and launch an effective evaluation one must first identify a training program. At present, this is not a huge issue. My experience with the initial implementation of SCORM within the DoD is that contracts are still being written per program. To conform to the current version of the SCORM, contractors are to my knowledge, developing programs as a complete content aggregation and calling complete lessons SCOs or breaking lessons into objectives and designating those chunks as SCOs. Evaluation, then, would be based on a complete content aggregation representing a training program. Forward thought must be given to the vision of the ADL Initiative though. If the plan is for learners to access learning objects individually to meet their specific learning needs, and this will change with the needs of each learner, what, then, will constitute a training program? Will such a thing exist? How will evaluation fit into such a vision? Will it? Shouldn’t it?

With the current SCORM taxonomy and the promise of individualized learning on the fly, anytime, anywhere, it would seem worthwhile to consider the evaluation of both programs represented by a content aggregation and of individual learning objects, if it is determined that their purpose is to provide instruction. Presuming that a content aggregation represents a training program for evaluation, the next step a designer must take is to determine the purpose of the evaluation. This will vary according to the program, the customer and the situation. Identifying the key stakeholders in the evaluation is important in both determining the purpose of the evaluation and in developing the key questions an evaluation should answer. From there, the evaluator would design an evaluation plan, develop data collection instruments and procedures, collect and analyze the data, and make use of the results according to the needs of the stakeholders (Russ-Eft & Preskill, 2001).

How does this process change when considering the evaluation of individual SCOs vice entire programs made up of learning objects? Evaluation of individual learning objects would take place as a part of the ISD process, much as any content piece would in a traditional training program in a typical ISD system. This isn’t program evaluation, but the line between content evaluation as a part of the ISD process and program evaluation becomes dangerously hazy with the undefined nature of a SCO. When is a SCO a piece of instruction and when is a SCO a program? Granularity itself and how a designer defines a learning object / SCO has a tremendous impact on how that object would be evaluated. Clearly only instructional objects could be measured according to Kirkpatrick’s levels. If the goal is to de-contextualize SCOs for maximum reusability, that also has an impact on how a learning object could be evaluated. Would one evaluate its effectiveness within its intended context? Or within any context? Or without context at all? “Evaluating the object in one context does not necessarily answer the question of how it performs in another context” (Williams, 2001, p. 188).

How does one evaluate at any of Kirkpatrick’s levels or within the guidelines of any evaluation model if learning is assembled differently and on the fly, combining instruction, performance support and information, for each learner? Current models of evaluation are ill-equipped to handle this modular, on-the-fly assembly of instructional programs. Still, with all the characteristics of traditional programs present even when assembled for the individual, it should be possible to establish a viable evaluation model. A model of evaluation needs to be developed for such an endeavor. This is an area where additional research and thought is greatly needed[L1] .

Service.

Service refers to how accessible e-learning is to learners (Rosenberg, 2001). The goal of the ADL Initiative is to enable anywhere, anytime learning. It is “preparing for a world where communications networks and personal delivery devices are pervasive and inexpensive, as well as transparent to the users in terms of ease of use, bandwidth and portability” (SCORM, 2001, p. 1-12). Clearly, we aren’t there yet and current accessibility varies widely across the DoD. For example, my experience has found access ranges from dial-up connections on some Marine Corps bases to high-speed internet connections on others. Evaluation of this element within the DoD could be accomplished through extensive surveying across DoD branches and divisions and even across contractors. By identifying the range of access within the DoD and between the DoD and contractors, a clearer picture could be assembled regarding service.

As mentioned previously, access in the context of the ADL Initiative can also refer to connection between content developers and learning content. The SCORM addresses accessibility through the content packaging specification that defines how SCOs should be packaged and labeled such that they can be located and accessed for use (SCORM, 2001).

Speed.

According to Rosenberg (2001), the speed of an e-learning initiative is a measure of three things: how quickly the e-learning can be up and running, how quickly the initiative can carry the content to the learner, and how fast content can be altered to meet the changing needs of the organization. Because the ADL Initiative has not moved yet beyond development of the SCORM, i. e. there is no content repository yet of reusable learning objects, many of the questions of speed remain unanswered. The hope is that with a large repository of reusable learning content, development will be instantaneous as depicted in the opening scenario of this paper. Second, the goal is for just-in-time delivery of training—immediate and on-the-fly assembly of content as needed for a learner anytime, anyplace. Again, this is as yet unrealized. Finally, with the eventual existence of a single online repository of content, the assumption is that update of material in that central location will ensure automatic and immediate update of material anywhere it appears (SCORM, 2001). Evaluation of the speed of the ADL Initiative, then, is not practically possible at this time. It could be done, however, once the appropriate infrastructure is in place and being used. Self-report by content developers on length of development time given a repository of existing content would provide useful measurement data. It will be of great interest to see if search time and attempts at locating useful content will in fact eclipse any potential savings in development time. Important in addition to actual time recordings, is perception as to whether the existence of a repository of content speeds development efforts. Speed in reaching the learner will perhaps be assessable by tracking the time of query of an object to the time of delivery to the learner. Some sort of tracking function would have to be enabled with each content object to gather this data. Perception by the learner is similarly important in this situation and would reveal useful data. Evaluation of update speed will require some sort of tracking function within a content repository that indicates content instances. In other words, when a content object is used, there should be some sort of tangible connection from the original object to its placement within a learning sequence. This would enable checks to assess whether updates to the original content are recorded in all instances of an object’s use.

Obviously the goals of the ADL Initiative won’t be realized overnight. It is very much a long term plan, depending on technological advances and the cooperation and collaboration of key partners. There is much promise, but it behooves the ADL Initiative to consider evaluation of its efforts along Rosenberg’s measures of cost, quality, service and speed to build a sound business case and address the concerns of critics like Greenagel (2002). Table 4 lists Rosenberg’s key criteria for building a sound business case, briefly describes what these criteria are, and summarizes the action taken by the ADL Initiative to address the key criteria.

Rosenberg’s Criteria	Criteria Explained	ADL Initiative Action
Cost.	The organization proves the ADL Initiative will save it money.	The SCORM emphasizes reuse and durability, arguing that these characteristics save development costs and programmer time, resulting in less DoD expenditure.
Quality.	The organization proves the ADL Initiative results in an increase in knowledge and improvement in performance.	Evaluation is not currently a part of dialogue surrounding ADL. Much to be considered.
Service.	The ADL Initiative is accessible.	Connectivity is currently not addressed but can be assessed. The SCORM goal of accessible content seeks to ensure access to learning content by content developers.
Speed.	The organization proves the ADL Initiative fosters responsiveness to the changing demands of the organization, its contractors and its employees.	The lack of maturity of the ADL Initiative prevents this from being measured. However, in preparation, tracking methods of learning content should be considered to enable assessment of update and use of content objects.

Table 4

Reinventing the Training Organization

If you think of a model of education as a wagon wheel, in the traditional model of education, the content and institution or classroom is at the center of the wheel. The learners rotate around that center, moving from place to place and instructor to instructor to access the content they need. The ADL Initiative, with its emphasis on just-in-time and just enough learning taking place anywhere, anytime, dramatically alters that model. Instead, in a distributed learning model, each learner becomes the center of his/her own wagon wheel, with the things that support learning rotating around that learner. This new model, a learner-centric model, allows more flexible access by the learner to people, content, and resources (Oblinger, 1996).

Rosenberg (2001) postulates that this new approach to learning can be quite difficult for traditional training organizations to accept and adjust to. Training functions operating under a retail model in which training is charged per user, for example, will need to move to a model in which training is based on well-defined business needs. That the ADL Initiative supports this non-traditional model is indicative of the necessary paradigm shift of the organization as a whole toward a new, required model of the training function. That much is encouraging, but a shift in thought and function by the organization does not guarantee a like phenomenon by its employees or contractors. Rosenberg (2001) acknowledges this by emphasizing that a re-skilling of the workforce may be required to accommodate this paradigm shift.

With learners at the center of any learning experience and with the new paradigm of learning objects, designers of instruction must dramatically change the way they look at learning and do their jobs. In the traditional model of education, instructional designers create courses, modules and lessons. In the new distributed learning model of education, instructional designers will be creating reusable, stand-alone chunks of learning content with accompanying descriptive metadata. The implications are significant.

Authoring learning objects.

Instructional Systems Design (ISD) traditionally occurs based on the ADDIE model—Analysis, Design, Development, Implementation, Evaluation. This is decidedly the case within the DoD. Critics of the traditional ISD ADDIE model see the recent developments in learning objects to be an opportunity to eliminate what they perceive to be an outdated, inflexible model.

“ISD, itself a linear and integrated process approach, mandates that designers know their target audience, write and sequence performance objectives, and then design the sequence of instruction, in that order. While the ISD approach seemed adequate for the days of rigid, boring and mundane computer-based training, it fostered a design strategy that is inadequate for a new e-learning market that rewards reuse and repurposing of content” (Hamel & Ryan-Jones, 2002, p. 3).

I couldn’t disagree more. According to the Department of Defense handbook on ISD / Systems Approach to Training and Education, the systems approach ensures that instructional designers accomplish the following activities:

Analyze missions, jobs, and tasks to determine instruction
Design instruction to meet the need
Develop instructional materials
Implement instructional materials
Evaluate the process and the product (Department of Defense, 2001).

Regardless of the product of such an effort, whether it is a set of distinct learning objects or a course of units and lessons, I believe firmly that the process is still a valid, workable one. In fact, in the development of instruction that must stand alone, contain minimal context and be delivered without an instructor, it is, in my opinion, even more important.

Cisco’s RIO project exemplifies how this methodology still works. Their process of creating performance-based learning objects follows a similar structure: first, the job task is identified, supporting skills and knowledge is identified, and training is developed in modular chunks to support the task (Downes, 2001). While the end product in this process is a series of learning objects instead of a traditional course or lesson, the fundamental method for ensuring a sound product are the same.

However, what does need to change in a traditional ISD approach, in addition to a modular instructional product, is the mindset throughout the process. “Design thinking needs to move from an approach that is oriented towards creating large integrated packages to one that is built around collections of specialized, reusable and granular components…object thinking needs to run throughout the process and not just in the construction phase” (Douglas & Shaffer, 2001, p. 5).

In an individualized and distributed learning environment, there may not be an instructor to rely on for clarification. Therefore, learning objects should be tied together with clear instructional strategies. These strategies should still be derived from a thorough analysis, including a needs analysis, learner analysis and task analysis when appropriate. A new wrinkle in the process, though, is what the Carnegie Mellon’s (2002) SCORM Best Practices Guide for Content Developers calls an “audience analysis.” Because any new content developed by an instructional designer should be reusable, this guide recommends that designers work with their team to brainstorm potential audiences of the content. This data will aid in the development of metadata so that others can appropriately reuse the materials.

A new first step in the design process is determining the content structure. Much like the traditional methods of sequencing, clustering and grouping objectives, the designer of new material must first determine the instructional strategy that will be used to deliver the material. Then, the designer may move on to something that should be kept in mind throughout the analysis process—determining the size of a SCO. In some cases, a SCO may be similar in scope and in nature to the content of a typical lesson (Downes, 2001). In others, it may represent a single instructional objective and supporting content. Regardless of size, a SCO should stand alone and be free of context (Carnegie Mellon, 2002). This characteristic gives many instructional designers heartburn, as many believe that context is king and to separate content from context is detrimental to learning. This is a new challenge and an area in need of additional research. Instructional designers must find a balance between providing enough context and not compromising the reusability of the resource.

In developing learning objects as instructional material, designers must now more than ever separate instructional content from display format. Because content may be reused in different contexts and displays, it is critical that other users be able to extract content from things like navigation, and what might be considered a “screen template.” This can be accomplished using cascading style sheets or XML Style Sheet Language (XSL) (Hamel & Ryan-Jones, 2002). Designers not familiar with these technical specifications will need to work closely with IT to ensure separation of content from form.

Authoring learning object metadata.

In the vision of the ADL Initiative, learning objects will be accumulated, tagged, and stored in repositories for broad distribution and use. Learning objects will be searchable based on metadata tagging for easy retrieval. Part of the responsibility of instructional designers in designing learning objects then will be in authoring the metadata for each object.

Metadata tagging is based on a scheme developed by the organization. A scheme or schema is the set of tags an organization selects to describe a learning object (Schatz, 2002). The scheme for the ADL Initiative is based on best practices from the key partners including the IMS Global Learning Consortium and IEEE LTSC and includes optional and mandatory metadata fields (SCORM, 2001). Though “the authoring of metadata itself will be straightforward for most course designers” (Downes, 2001, p. 23) and may be automated in many cases, designers would be wise to have an understanding of the ADL Initiative metadata schema and completion of metadata fields.

Reusing learning objects.

Instructional designers will not just be responsible for creating new objects, tagging them, and submitting them to a learning object repository. Once objects are readily available for reuse, designers will have available to them a new resource—a bank of learning objects for their own reuse. Designers, then, must learn to access repositories and search for reusable content that may be used or revised to meet new requirements (Douglas, 2001). In tracking the update and modification of original objects, again, some consideration must be given to how that tracking will take place within the object and whose role it will be to update the object.

Evaluating learners.

Under the current SCORM model, testing for learning objects is not well defined. Designers will have to determine to what level they wish to track student responses to test items. LMSs are what allow test scores to be tracked and attributed to a learner, but a SCO represents the lowest level of granularity that an LMS can track. That means that if an assessment is built into a learning object within the same SCO, the LMS will be able to report back a pass/fail status or single score, but no detailed information. The same situation is true if the designer builds the assessment as a separate SCO. In terms of determining test reliability and validity and gathering useful formative or summative evaluation data, neither option is ideal. In order to track a learner’s response to individual assessment items, a designer would be forced to make each test item a SCO, group the SCOs together into an aggregation, and have the LMS track each learner response (Carnegie Mellon, 2002). This has important implications for the amount of work involved in creating useful assessments of learning objects and for evaluating a learner. Content developers will need to address evaluation during and after SCO development much as they would during the ISD process in designing and developing traditional learning content so that performance criteria and conditions are either a part of the SCO or somehow otherwise “attachable” to a SCO within the metadata.

Program evaluation.

Designers of instruction using learning objects will continue to be responsible for determining program goals and objectives, developing content, and structuring the training using SCOs and content aggregations. A final task of determining the achievement of results by learners is a critical issue designers still face. Evaluation of learning programs is an often overlooked, but integral part of any training / learning effort (Russ-Eft & Preskill, 2001). Whether the effort is spearheaded by learning object designers or internal / external evaluators tapped for that specific purpose, the need remains. Within the DoD, the most likely scenario is that program evaluation will continue to fall to the content developers as part of a contract. It is of particular importance to evaluate programs incorporating new techniques or strategies, that are critical or important to an organization, and about which the organization has many questions. With the implementation of the ADL Initiative and use of learning objects in training, all SCORM-conformant learning programs within the DoD now meet all three criteria. Clearly, modular SCO construction will be new, both to designers and to the customer; the ADL Initiative and its subsequent efforts associated with the SCORM are important to the DoD; and with the abstract nature of the SCORM, there are many unanswered questions about its real-life application in DoD contracts.

Clearly, there are significant changes in store for content developers with the implementation of the ADL Initiative. New skills in working with learning objects will undoubtedly be required. It could be argued that it is beyond the domain of the DoD to re-skill content developers who may be employed by outside organizations. The problem with that theory is that no organizations can build competencies in employees without a better defined plan for how work will be done under the ADL Initiative. A recent study by Stout, Slosser and Hays (2002) sponsored by the Joint ADL Co-lab illustrates much of the confusion associated with content development at this stage in the ADL Initiative’s development. Key complaints from workers attempting to implement SCOs and produce SCORM-conformant learning material include: ambiguous standards, a confusion regarding what constitutes a SCO, lack of guidance on proper metadata tagging, and handling firewall and security issues. How do organizations instill the needed competencies to design learning content for the DoD when there is no clear or established way to do that? How does a content developer create reusable SCOs when s/he doesn’t understand how reusability will work and has no clear idea of the definition of a SCO? How does a content developer complete metadata tagging when s/he doesn’t understand the metadata fields and their purpose or the scheme for doing so? How do you effectively evaluate a learning object or a content aggregation if there is no applicable evaluation model for a learning object environment? Without answers to these and similar questions, competence necessary to develop learning content for the DoD simply cannot be achieved.

With the momentum of the ADL Initiative, the DoD demonstrates a commitment to an entirely new model of training, from an instructor-centered model to a learner-centered one. This is an important first step, according to Rosenberg, in reinventing the training function. Also critical though, is re-skilling the workforce jointly as these changes take place to ensure implementation is successful. In this area, the DoD falls a bit short. Table 5 lists Rosenberg’s key criteria for reinventing the training function, briefly describes what these criteria are, and summarizes the action taken by the ADL Initiative to address the key criteria.

Rosenberg’s Criteria	Criteria Explained	ADL Initiative Action
Change in model.	The training function moves from a retail model to a model in which training aligns with organizational goals.	The ADL Initiative supports a learner-centric model vice an instructor-centric one.
Re-skill the workforce.	With this paradigm shift comes a requirement for new competencies. The organization must attend to those staff development needs.	Much work remains in better defining the SCORM standards and guidelines for content developers so that competencies can be appropriately built.

Table 5

Managing Change and Developing a Learning Culture

The ADL Initiative envisions a future of just-in-time, just enough learning, accessible anytime, anywhere, to all learners in the DoD. It anticipates enabling this type of learning through the development of a learning object economy made up of sharable, reusable learning content objects. This is a fundamental shift in the way online training and all other learning takes place within the Department of Defense, and indeed, within the educational community as a whole. Change can be painful and often elicits resistance on the part of those who will be affected by it. Rosenberg (2001) suggests that in order for e-learning to thrive, attention must be paid to the four “Cs”: culture, champions, communications and change.

Culture.

Building a learning culture is about creating enthusiasm and support for learning and recognizing that learning is a valued part of what people do. Doing this effectively means moving from a training mindset in which just-in-case training is “pushed,” to a “pull” mentality. This is the basic tenet of the ADL Initiative. But it’s a new way of looking at things for many in the training industry. Part of successfully changing that mindset and realizing the goals of the ADL Initiative means establishing a culture that fosters and supports that change. Rosenberg (2001) believes that accountability is also part of the equation. By making superiors and—in the case of the DoD—contractors accountable for learning, they become directly invested in the outcome. In the eight training contracts I’ve supported in my work as an ISD with a government contractor, only two have incorporated a program evaluation plan. In one of those projects, a decades old plan was recycled over and over to meet the requirement but had no application to the project and was not implemented. Simply, if content developers and supervisors within the project have no obligation to ensure that learning takes place as a result of their efforts, they can detach themselves from the learning climate and avoid accountability. By ensuring that each contract mandates an evaluation plan and that results, good or bad, are shown and dealt with, the DoD could go a long way towards getting supervisors and content developers on board with support for learning.

Still, mandates are not enough. Reward and motivation are critical components in building culture (Rosenberg, 2001). Much has already been said in this paper about the importance of a reward system to motivate the participation of content developers. It is not enough to build it and assume they’ll come. Research has proven that they won’t—not without sufficient motivation (Davenport & Prusak, 1998, Rosenberg, 2001, Stewart, 1997, Tobin, 1998).

Champion.

Part of building a learning culture, too, is having the support of organizational leaders. Top organizational support is demonstrated by adequate funding and interest in an initiative. The ADL Initiative is clearly supported by leaders in the DoD and collaborative partners in academia and industry. Still, there is no real defined key leader. On the ADL web site (2003), each discussion forum has a moderator who answers questions and stimulates collaborative discussion surrounding key issues central to the ADL Initiative. But neither moderator represents a voice of the ADL beyond the boundaries of the forum. SCORM is a hot topic on training and development listserves, including trdev-L, e-learningleaders, and DEOS-L (2002). There is one outspoken leader representing the ADL Initiative with decisive strength. Mark Oehlert, deputy Director for Communications for the ADL Co-Lab in Alexandria, VAhas worked on the ADL Initiative for almost 5 years and can be considered well versed in the development, history and aims of ADL and SCORM (personal communication, 2002). Mr. Oehlert has led a good public charge for the SCORM specifically and ADL Initiative broadly. But the effort would benefit from the support from more top voices, speaking loudly and clearly, to emphasize the effort is “durable enough to build the momentum and critical mass that’s necessary to transform the organization into one that accepts e-learning, and learning in general, as a natural part of the firm’s everyday work life” (Rosenberg, 2001, p. 189).

Communication.

With leadership in place, developing an environment that supports an e-learning initiative is facilitated by effective communication. The ADL Initiative has taken steps to make its development an open process. Plugfests encourage participation and collaboration around the development of the SCORM. The ADL web site (2003) provides links to news and events surrounding the initiative, discussion forums for technical developers, instructional designers, and content developers, and resources related to the initiative. These are all good steps in developing a communication plan that support communications about and surrounding the ADL Initiative (Rosenberg, 2001).

Change.

The ADL Initiative asks people to do things differently. It asks learners and those dedicated to helping people learn to think about learning in a different way. It asks learning content developers to approach their jobs from a completely new perspective. Much of the immediate impact of the ADL Initiative and SCORM is transparent to the learner. It is a much more significant change, however, for those who design the learning experiences. Building a learning culture, identifying champions of the initiative, and facilitating open communication are all necessary in order to successfully manage this change. Still, there is much to be done to ensure that those impacted by the ADL Initiative are ready to accept that impact. Change management refers to a systematic change strategy that ensures that the people in an organization are committed and able to execute a business plan. Rosenberg asserts that change management happens along three dimensions: motivation, competence and resources (2001). All of these elements have been explored and discussed in detail thus far in this paper. Clearly, they represent areas of weakness for the DoD and bear further attention. Not only are they necessary within the context of building a culture of learning, and reinventing the training organization, motivation, competence and access to resources represent the fundamental building blocks of supporting the critical change brought about by the ADL Initiative.

Mandating change is not enough. In order for the ADL Initiative to fully enjoy adoption and success, a culture that values learning must be nurtured. This significant change for learners and learning content developers must be recognized and addressed through effective and planned change management (Rosenberg, 2001). Table 6 lists Rosenberg’s key criteria for building a learning culture and managing change, briefly describes what these criteria are, and summarizes the action taken by the ADL Initiative to address the key criteria.

Rosenberg’s Criteria	Criteria Explained	ADL Initiative Action
Accountability.	The organization holds contractors and learner supervisors accountable for training results.	DoD contracts typically fail to emphasize evaluation plans. New ones should.
Reward participation.	The organization should incent and reward efforts that contribute to building a learning culture.	Certification programs help but are insufficient.
Identify champions.	The organization has visible and supportive champions of the e-learning cause who can rally support.	Not clearly defined within the DoD.
Supports motivation, competence and resources.	The organization motivates those affected by change, instills competence, and provides necessary resources.	DoD provides insufficient motivation to learning content developers, SCORM is ill-defined.

Table 6

Summary and Recommendations

This paper has sought to evaluate the ADL Initiative against key criteria identified by Rosenberg’s (2001) strategic foundation for learning. Table 7 provides a summary list of the outcome of the evaluation along the 6 key components for creating a successful e-learning strategy: infrastructure, online training and Knowledge Management, learning architecture, sound business case, reinventing the training organization, and learning culture, management ownership and change management.

Infrastructure
Rosenberg’s Criteria	Criteria Explained	ADL Initiative Action
Selection of a common LMS / has a goal of interoperability.	A single LMS is selected to ensure portability of learning content within the organization.	SCORM standards ensure interoperability of learning content across LMSs.
Learning content is accessible.	Users have adequate connectivity and can access learning content.	Connectivity of content developers is beyond the scope of the DoD. SCORM standards ensure content accessibility.
Single portal entry.	A single portal for access and retrieval of e-learning content is available for users.	SCORM documentation seems to speak to a single content repository—but it is as yet undefined.
Online Training & Knowledge Management
Rosenberg’s Criteria	Criteria Explained	ADL Initiative Action
Generating support through demonstration.	The organization shows how the system works.	Plugfests allow participants to test their courseware and LMS for SCORM conformance for inclusion in a DoD content repository.
Developing an oversight group.	The organization provides management of content and establishment of common policy, guidelines and standards enhance the usability of the system.	None known at this time beyond establishment of the 3 Co-laboratories.
Building in collaboration.	The organization provides an opportunity for users to collaborate with one another.	ADL web site has forums for asynchronous communication.
Providing incentive and reward for participation.	The organization incents users to share information and content and follow the guidelines and policies established.	None other than contract mandates for “SCORM-conformance.”
Developing a knowledge structure.	There are clear logical links and tags between content elements so that content can seamlessly and naturally be assembled as needed.	Metadata tagging is available, but poorly defined SCOs make combination and sequencing a potential mine field for developers.

Learning Architecture
Rosenberg’s Criteria	Criteria Explained	ADL Initiative Action
Conducting an analysis, basing design decisions on competencies, business needs and stakeholder input.	The organization makes decisions on learning design based on factors such as audience needs, business need and learning domain.	Currently, the ADL Initiative focuses on web-based solutions to the exclusion of all others. Many DoD contracts pre-suppose the web as the delivery media.
Reuse existing content.	The organization avoids reinvention of the wheel by reusing or repurposing existing learning material when possible.	The SCORM supports and facilitates the reuse of learning content with specific standards on reusability.
Use web as unifying portal for learning solutions; create community online; engage learners.	The organization takes steps to create a learning environment on the web that supports the blended approach to learning.	Currently, the ADL Initiative focuses on web-based solutions to the exclusion of all others. Many DoD contracts pre-suppose the web as the delivery media.
Sound Business Case
Rosenberg’s Criteria	Criteria Explained	ADL Initiative Action
Cost.	The organization proves the ADL Initiative will save it money.	The SCORM emphasizes reuse and durability, arguing that these characteristics save development costs and programmer time, resulting in less DoD expenditure.
Quality.	The organization proves the ADL Initiative results in an increase in knowledge and improvement in performance.	Evaluation is not currently a part of dialogue surrounding ADL. Much to be considered.
Service.	The ADL Initiative is accessible.	Connectivity is currently not addressed but can be assessed. The SCORM goal of accessible content seeks to ensure access to learning content by content developers.
Speed.	The organization proves the ADL Initiative fosters responsiveness to the changing demands of the organization, its contractors and its employees.	The lack of maturity of the ADL Initiative prevents this from being measured. However, in preparation, tracking methods of learning content should be considered to enable assessment of update and use of content objects.

Reinventing the Training Organization
Rosenberg’s Criteria	Criteria Explained	ADL Initiative Action
Change in model.	The training function moves from a retail model to a model in which training aligns with organizational goals.	The ADL Initiative supports a learner-centric model vice an instructor-centric one.
Re-skill the workforce.	With this paradigm shift comes a requirement for new competencies. The organization must attend to those staff development needs.	Much work remains in better defining the SCORM standards and guidelines for content developers so that competencies can be appropriately built.
Learning Culture, Management Ownership, and Change Management
Rosenberg’s Criteria	Criteria Explained	ADL Initiative Action
Accountability.	The organization holds contractors and learner supervisors accountable for training results.	DoD contracts typically fail to emphasize evaluation plans. New ones should.
Reward participation.	The organization should incent and reward efforts that contribute to building a learning culture.	Certification programs help but are insufficient.
Identify champions.	The organization has visible and supportive champions of the e-learning cause who can rally support.	Not clearly defined within the DoD.
Supports motivation, competence and resources.	The organization motivates those affected by change, instills competence, and provides necessary resources.	DoD provides insufficient motivation to learning content developers. SCORM is ill-defined.

Table 7

The bulk of the effort of the ADL Initiative has been on the development of the SCORM, and it shows. The Initiative scores high marks on building a strong infrastructure that clearly addresses interoperability issues. Still, the SCORM is not yet well-defined, particularly with respect to SCO combination and granularity. This knowledge structure weakness has a direct impact on many other strategic areas, including incentive, motivation and reward, and competence of learning content developers. This must be addressed to lessen the impact of this significant change in the way of doing business and ensure the success of the Initiative as a whole. By better defining SCOs, and recognizing the value and differentiation of information vice instructional SCOs, this barrier may be overcome.

Another significant weakness of the ADL Initiative is evaluation. Experience shows that the DoD can be accused of failing to emphasize program evaluation in its pre-ADL contracts. But with the changes associated with modular design using learning objects, this issue becomes even more problematic. The development of a model of evaluation for this type of design is something warranting further research and attention.

Overall, it is in the areas of reinvention of the training function and in managing this change by developing a learning culture that the DoD faces the most significant challenges. This isn’t surprising. The early review of the paradigm shift faced by training functions today clearly illustrates what a fundamental change this is. Simply, it isn’t easy. But it is required for e-learning success.

There is no doubt that in order for the ADL Initiative to succeed, a standard, solid infrastructure must be in place. The emphasis on the part of the ADL Initiative on development and refinement of the SCORM is both necessary and understandable. Clearly, there is work left to be done in establishing an infrastructure that supports the development, storage, retrieval and reuse of learning objects. With technology increasing with lightening speed every day, there is a need for diligence in keeping the SCORM up to date with the capabilities available. However, the ADL Initiative focuses on technology to the exclusion of all other things at its peril. There is much left to be resolved.

On the surface, the ADL Initiative seems to be a well-conceived effort with a clearly defined mission and set of measurable, achievable goals. But when examined against the criteria presented by Rosenberg for a successful e-learning strategy, some areas for improvement are revealed. With a single-track focus on infrastructure, the ADL Initiative has failed so far to investigate and invest in the development of new approaches to learning, a learning architecture, and culture. If the DoD is to reinvent itself as a learning organization as it must to ensure a successful e-learning strategy, there is much work to do.

References

Advanced Distributed Learning. (2003). Academic ADL co-lab. Retrieved December 1, 2002

from the Advanced Distributed Learning Web site:

http://www.adlnet.org/index.cfm?fuseaction=colabacd&cfid=125178&cftoken=26890577

Advanced Distributed Learning. (2003). Alexandria ADL co-lab. Retrieved December 1, 2002

from the Advanced Distributed Learning Web site:

http://www.adlnet.org/index.cfm?fuseaction=colabadl&cfid=125178&cftoken=26890577

Advanced Distributed Learning. (2003). Joint ADL co-lab. Retrieved December 1, 2002

from the Advanced Distributed Learning Web site:

http://www.adlnet.org/index.cfm?fuseaction=colabjnt&cfid=125178&cftoken=26890577

Academic Advanced Distributed Learning Co-Lab. (2002). ADL: Intro to ADL and the SCORM

(online tutorial). Retrieved December 15, 2002 from the Academic Advanced

Distributed Learning Co-Laboratory Web site:

http://www.academiccolab.org/learn/index.html

Advanced Distributed Learning Initiative. (2001). Sharable Content Object Reference Model

(SCORM) version 1.2. Retrieved July 12, 2002 from the Advanced Distributed

Learning Co-Laboratory Web site:

http://www.adlnet.org/index.cfm?fuseaction=scormdown&cfid=351048&cftoken=86366418

Alexander, S. (2001) E-learning developments and experiences. Education + Training, 43

(4/5), 240-248.

Barnes & Noble (2003). Search results: web-based training. Retrieved February 23, 2003 from

the World Wide Web:

http://search.barnesandnoble.com/booksearch/results.asp?WRD=web%2Dbased+training&userid=2WVRKW6SKH

Barron, T. (2000, March). Learning object pioneers. Learning Circuits. Retrieved August 15,

2002 from the World Wide Web: http://www.lib.usm.edu/~instruct/guides/apa.html

Beer, V. (2000). The web learning fieldbook: Using the World Wide Web to build workplace

learning environments. San Francisco: Jossey-Bass/Pfeiffer.

Book, E. (2001). “Plugfests” help standardize online learning technology. National Defense

Magazine, 1 (12). Retrieved November 5, 2002 from the World Wide Web:

http://www.nationaldefensemagazine.org/article.cfm?Id=672

Bowman, S. (1998). How to give it so they get it. Glenbrook, NV: Author.

Broadbent, B. (2002). ABCs of e-learning: Reaping the benefits and avoiding the pitfalls. San

Francisco: Jossey Bass/Pfeiffer.

Cantor, J. (1992). Delivering instruction to adult learners. Dayton, OH: Wall & Emerson.

Carnegie Mellon Learning Systems Architecture Lab (2002). SCORM best practices guide for

content developers, version 1.7. Retrieved December 1, 2002 from the Learning Systems

Architecture Lab Web site: http://www.lsal.cmu.edu/lsal/expertise/projects/developersguide/developersguide/guide-v07-01102002.pdf

Clark, R.C. (1998). Recycling knowledge with learning objects. Training and Development, 52

(10), pp. 60-64.

Cohen, S. & Payiatakis, D. (2002). E-learning: Harnessing the hype. Performance

Improvement, 41(2), 7-15.

Davenport, T. & Prusak, L. (1998). Working knowledge: How organizations manage what they

know. Boston: Harvard Business School Press.

Department of Defense. (2003). DoD 101. Retrieved December 21, 2002 from the World Wide

Web: http://www.defenselink.mil/pubs/dod101/

Department of Defense. (2001). MIL-HDBK-29612-2A Department of Defense handbook:

Instructional systems development / systems approach to training and education.

Retrieved November 13, 2002 from the World Wide Web: http://www.dtswg.org/PDF%20Files/-2%20hdbk.pdf

Douglas, I. (2001). Instructional design based on reusable learning objects: Applying lessons of

object-oriented software engineering to learning systems design. Proceedings of the 31^st

ASEE/IEEE Frontiers in Education Conference.

Douglas, I. & Schaffer, S. (2001). A methodological framework for an object-oriented

performance support system. Learning Systems Institute: Florida State University.

Downes, S. (2001). Learning objects: Resources for distance education worldwide. International

Review of Research in Open and Distance Learning, 2 (1).

Driscoll, M. (1998). Web-based training: Using technology to design adult learning

experiences. San Francisco: Jossey-Bass/Pfeiffer.

Driscoll, M. (2002). Web-based training: Creating e-learning experiences. San Francisco:

Jossey-Bass/Pfeiffer.

Drucker, P. (1994). Post-capitalist society. New York: HarperCollins.

Gagné, R., Briggs, L. &Wager, W. (1992). Principles of instructional design. Fort Worth:

Harcourt Bruce Jovanovich College.

Gagné, R. & Medsker, K. (1996). The conditions of learning. Fort Worth: Harcourt Bruce

Jovanovich College.

Galvin, T. (2002). 2002 industry report. Training, 39 (10), 24-73.

Gates, B. (2000). Business @ the speed of thought: Using a digital nervous system. New York:

Warner Books.

Greenagel, F. (2002, July). The illusion of e-learning: Why we are missing out on the promise

of IP technology. E-learning Magazine. Retrieved February 2, 2003 from the World

Wide Web: http://www.elearningmag.com/elearning/article/articleDetail.jsp?id=26850

Hamel, C. & Ryan-Jones, D. (2002). We’re not designing courses anymore. Retrieved

December 31, 2002 from the World Wide Web: http://www.fcful.dk/artikel/hamel.htm.

Harris, P. (2002, November). How the U. S. military is reinventing learning. Learning Circuits.

Retrieved December 5, 2002 from the World Wide Web:

http://www.learningcircuits.org/2002/nov2002/harris.html

Institute of Electrical and Electronics Engineers, Inc., Learning Technology Standardization

Committee. (2001). Draft standard for learning object metadata. Retrieved November

13, 2002 from the World Wide Web: http://ltsc.ieee.org/wg12/LOM_WD6.pdf

Jorgensen, E., Fuller, J., Post, T. & Rainey, S. (1995). Potential benefits to Navy training

programs resulting from increased use of Interactive Electronic Technical Manuals.

Status Report on Draft Specifications and Handbooks Developed by the Tri-Service Working Group for IETMs. NSWCCARDDIV-TR-96/016. Retrieved December 15, 2002 from the World Wide Web: http://navycals.dt.navy.mil/ietm/Phase1.pdf

Kaufman, G. & Svitak, A. (2002, July). Transforming training. Defense News. Retrieved

August 30, 2002 from the World Wide Web:

http://www.adlnet.org/ADLDOCS/Documents/Defense%20News.doc

Kilby, T. (2001). The direction of web-based training: A practitioner’s view. The Learning

Organization, 8(5). Retrieved February 28, 2003 from the World Wide Web:

http://www.webbasedtraining.com/archive_direction.aspx.

Kraan, W. & Wilson, S. (2002). SCORM is not for everyone. Retrieved December 12, 2002

from the Centre for Educational Technology Interoperability Standards (CETIS) Web

site: http://www.cetis.ac.uk/content/20021002000737

Marquardt, M. & Keasley, G. (1999). Technology-based learning: Maximizing human

performance and corporate success. Washington, D. C.: St. Lucie.

Masie Center. (2002). Making sense of learning specifications and standards: A decision

maker’s guide to their adoption. Retrieved July 23, 2002 from the World Wide Web:

http://www.masie.com/standards/S3_Guide.pdf

Merriam-Webster. (1990). Webster’s ninth new collegiate dictionary. Massachusetts: Author.

Merrill, D., Drake, L., Lacy, M., Pratt, J. & ID2 Research Group (1996). Reclaiming

instructional design. Educational Technology, 36(5), 5-7.

Mills, S. (2002). Learning about learning objects with learning objects. Retrieved November

13, 2002 from the AliveTek, Inc. Web site:

http://www.alivetek.com/learningobjects/site_paper.htm

Moncarz, R. (2002). Training for techies: career preparation in information techology.

Retrieved February 15, 2003 from the Bureau of Labor Statistics Web site:

http://www.bls.gov/opub/ooq/2002/fall/art04.pdf

Netlingo™ (1995). Retrieved December 20, 2002 from the World Wide Web:

http://www.netlingo.com

Noe, R. (1999). Employee Training & Development. Boston: Irwin McGraw-Hill.

Oakes, K. & Rengarajan, R. (2002). An objective view of learning objects. Training and

Development, 56 (5), pp. 103-105.

Oblinger, D.G. and Maruyama, M.K. (1996). Distributed learning. (CAUSE Professional Paper

Series, No. 14.) Boulder, CO: CAUSE.

O’Driscoll, T. (1999). Achieving desired business performance: A framework for developing

Human Performance Technology in organizations. Washington, D. C.: International

Society for Performance Improvement.

Quinn, C. and Hobbs, S. (2000). Learning objects and instruction components. Educational

Technology & Society 3 (2), 13-20.

Recker, M. M. & Wiley, D. A. (2001). A non-authoritative educational metadata ontology for

filtering and recommending learning objects. Journal of Interactive Learning

Environments. The Netherlands: Swets & Zeitlinger. Retrieved September 27, 2002

from the World Wide Web: http://wiley.ed.usu.edu/docs/non-authoritative.pdf

Rosenberg, M. (2001) E-Learning: Strategies for delivering knowledge in the digital age. New

York: McGraw-Hill.

Rothwell, W. (1996). Beyond training and development. New York: American Management

Association.

Rothwell, W. & Kazanas, H. (1998). Mastering the instructional design process: A systematic

approach. San Francisco: Jossey-Bass/Pfeiffer.

Rummler, G. (1999). Transforming organizations through Human Performance Technology. In

H. D. Stolovitch & E. Keeps (Eds.) Handbook of Human Performance Technology (pp.

47-66). San Francisco: Jossey-Bass/Pfeiffer.

Russ-Eft, D. & Preskill, H. (2001). Evaluation in organizations. Cambridge, MA: Perseus.

Schatz, S. (2001). Paradigm shifts and challenges for instructional designers: An introduction to

meta tags and knowledge bits. Retrieved July 10, 2002 from the IMS Global Learning

Consortium, Inc. Web site: http://www.imsproject.org/feature/kb/index.html

Shook, W., Dargue, B., & Carlton, T. (2002) ADL/SCORM where do we go from here? The

integration of training and technical data. Proceedings of ITEC 2002.

Silberman, M. (1998). Active training. San Francisco: Jossey-Bass/Pfeiffer.

Smith, E. (2002, October/November). Much ado about SCORM. Training and Simulation, pp.

31-35.

Stout, R., Slosser, S. & Hays, R. (2002). Sample lessons learned from Advanced Distributed

Learning efforts. Retrieved December 1, 2002, from Joint Advanced Distributed

Learning Co-Laboratory Web site:

http://www.jointadlcolab.org/Stout_LessonsLearnedDoc.doc

Stewart, T. (1997). Intellectual Capital: The New Wealth of Organizations. New York:

Doubleday.

Tinker, R. (2001). E-learning quality: The concord model for learning from a distance.

Bulletin, 85 (628).

Thornborough, B. (2001). Knowledge management: General approaches respecting the

ongoing development of The Learning Manager (TLM). Retrieved December 5, 2002

from the Learning Management Corporation Web site:

http://www.thelearningmanager.com/pubdownloads/Knowledge_Management.pdf

Tobin, D. (1998). The knowledge-enabled organization: Moving from “training” to “learning”

to meet business goals. New York: American Management Association.

Toracco, R. (1999). Integrating learning with working: A reconception of the role of workplace

learning. Human Resource Development Quarterly, 10 (3), 249-270.

Tyler, K. (2002). Take e-learning to the next step. HR Magazine, 47 (2), 56-61.

Weaver, P. (2002, August). Preventing e-learning failure. Training & Development, pp. 45-50.

Weintraub, R. & Martineau, J. (2002, June). The just-in-time imperative. Training &

Development, pp. 50-58.

Welsch, E. (2002, August) SCORM: Clarity or calamity? Online Learning Magazine.

Retrieved August 5, 2002 from the World Wide Web:

http://www.onlinelearningmag.com/training/search/search_display.jsp?vnu_content_id=1526769

Wiley, D. A. (2000). Learning objects and the new CAI. Retrieved October 20, 2002 from the

World Wide Web: http://wiley.ed.usu.edu/docs/instruct-arch.pdf

Wiley, D. A. (2000). Learning object design and sequencing theory. Dissertation Retrieved

December 13, 2002 from the World Wide Web:

http://wiley.ed.usu.edu/docs/dissertation.pdf

Wiley, D. A. (2001). Connecting learning objects to instructional design theory: A definition, a

metaphor, and a taxonomy. In D. A. Wiley (Ed.) The instructional use of learning

objects (pp. 3-23). Bloomington, Indiana: Agency for Instructional Technology.

Wiley, D. A., Gibbons, A. & Recker, M. M. (2000). A reformulation of learning object

granularity. Retrieved October 20, 2002 from the World Wide Web:

http://reusability.org/granularity.pdf

Williams, D. (2001). Evaluation of learning objects and instruction using learning objects. In D.

A. Wiley (Ed.) The instructional use of learning objects (pp. 3-23). Bloomington,

Indiana: Agency for Instructional Technology.

Zemke, R. (1998). Wake up! (And reclaim instructional design). Training, 35 (6), pp. 36-42.