Learning journeys for scalable AI education: an MIT - USAF collaboration

50th Annual Conference in September 2022 1529 Andres Felipe Salazar Gomez, Aikaterini Bagiati, Cynthia Breazeal, John Radovan and Kathleen D Kennedy Learning Journeys for Scalable AI Education: An MIT - USAF Collaboration - 10.5821/conference-9788412322262.1189 doi:10.5821/conference-9788412322262.1189 LEARNING JOURNEYS FOR SCALABLE AI EDUCATION: AN MIT - USAF COLLABORATION A. Bagiati1 Massachusetts Institute of Technology Cambridge, MA, USA ORCID 0000-0003-4238-2185 A.F. Salazar-Gomez Massachusetts Institute of Technology Cambridge, MA, USA ORCID 0000-0003-3749-6815 J. Radovan United States Air Force Cambridge, MA, USA N/A K. D. Kennedy Massachusetts Institute of Technology Cambridge, MA, USA ORCID 0000-0002-4252-4836 C. Breazeal Massachusetts Institute of Technology Cambridge, MA, USA ORCID 0000-0002-0587-2065 Conference Key Areas: Curriculum development, AI in education Keywords: AI curriculum development, training in AI, diverse learners, skills development, scalability ABSTRACT In 2021 the United States Air Force (USAF) and the Department of Defence (DoD) entered into a collaboration with multiple units within the Massachusetts Institute of 1 Corresponding Author A. Bagiati [email protected]u 50th Annual Conference in September 2022 1530 Technology (MIT) to develop a new academic program focusing on Artificial Intelligence (AI) training. Given the size and the diversity within the body of USAF employees, the goal of this collaboration is to design and implement an innovative program that will achieve maximum learning outcomes at scale for learners with diverse roles and educational backgrounds. This program is now piloting and evaluating three different learning journeys addressing three different groups of USAF employees (USAF leaders and decision makers; technology developers; and daily frontend technology users). The learning journeys were designed based on each group’s specific professional needs and academic backgrounds, and they include combinations of online synchronous and asynchronous courses and face-to-face activities. The program’s pilot is currently underway and evaluation research findings are informing the next program iterations. The ultimate goal of this program is to formulate general recommendations on how to serve large numbers of diverse learners at scale in an optimum way. In addition to an evaluation pilot study, MIT experts on AI and the Science of Learning have been asked to review the program and their feedback will be integrated into the next program iteration. This paper presents the three learning journeys as originally designed to serve the three first diverse cohorts of learners, as well as the plan for future improvement and implementation of the program. 1 INTRODUCTION Artificial Intelligence (AI) is a rapidly growing field expected to change the way organisations and businesses operate, as well as how grand challenges of the future will get tackled [1]. As the AI field progresses the number of jobs in or in conjunction with this sector is also growing. Industries need upskilling programs for their employees, “universities are clamoring to create curriculum that can meet the demand for computer scientists with AI expertise”, and governments are working to find ways to best educate the general public [2]. Approaches to AI education vary significantly. They include topics ranging from understanding of fundamental concepts, applications and solutions, to development of hard-core technical knowledge like machine learning, robotics and big data analytics. Technical knowledge is guided mainly by the engineering and computer science world, but there is an urgent call for timely discussions about societal and ethical implications of AI development and use [1]. Despite the urgent need for State-of-the-Art AI education, this field is at a nascent stage and there is still a lack of understanding regarding what the optimum ways would be to approach different professionals with different needs, along with the education of the general public. Within this context the US Department of Defence (DoD) decided it is absolutely crucial to harness the potential of AI to positively transform all functions of the Department, while also supporting AI education related research and development targeting a broader body of learners. Realising this vision requires identifying 50th Annual Conference in September 2022 1531 appropriate use cases for AI across DoD, rapidly piloting solutions, and scaling successes across the whole enterprise [3]. The Joint Artificial Intelligence Center (JAIC) was the focal point selected for carrying out this vision [3]. As careful AI adoption in all operations would require both State-of-the-Art technology and a highly educated and trained body of employees it became clear early on that the DoD would need to closely collaborate with academia or other centers of innovation providing expertise on AI education, to design the appropriate educational resources and training programs [4]. Given the size of the United States Air Force (USAF) workforce (~680.000), the difference in employees educational (high school diplomas to graduate degrees), age (18-60), cultural backgrounds, and the difference in the nature of their daily working needs and operations (full spectrum of officer and enlisted ranks), in 2020 DoD decided to collaborate with multiple units of the Massachusetts Institute of Technology (MIT) and fund an AI Education research project that would explore the development of a set of different AI learning journeys targeting subgroups of diverse learners (for this paper each group will be called an archetype). Goal of this mixed-method research approach is to understand the learner experience regarding content, pedagogy, and educational technologies engaged in the learning process. Findings are expected to guide further iterations of the program as the number of learners will be scaling up. This concept paper will present the development and implementation of the first 3 learning journeys that started in the summer of 2020, and a plan for future improvement. 2 AI EDUCATION PROGRAM DEVELOPMENT 2.1 Background To start this endeavour JAIC engaged in a primary analysis of the USAF body of employees and presented a set of 6 archetypes (see Table 1), as well as desired AI related needs, skills and competencies for each one of them. A description and analysis regarding desired AI competencies and learning outcomes per archetype, as well as expected levels of understanding can be found in greater detail at the 2020 DoD AI Education Strategy [4]. Table 1. Learner archetypes (Modified from [4]) Name Descriptions Lead AI Decides policy and doctrine, including how AI tools can or will be used; builds AI vision and plan Drive AI Ensures appropriate AI tools and capabilities are developed and delivered Create AI Creates AI tools to meet current and future needs Embed AI Embedded with Employ AI, establishes AI systems and provides end-user support at tactical edge 50th Annual Conference in September 2022 1532 Facilitate AI Represents user to ensure appropriate AI tools are developed and delivered to address use cases Employ AI End-user of AI tools, provide feedback on and requirements for AI tools. 2.2 USAF - MIT AI Accelerator In 2019 the Department of the Air Force (DAF) signed a cooperative agreement with MIT and jointly created the Artificial Intelligence Accelerator, hosted at MIT. In summer 2020 the USAF-MIT AI accelerator, involving staff from MIT Campus, MIT Lincoln Laboratory, the DAF, along with STEM education researchers from MIT Open Learning and the MIT Media Lab, launched the “Know-Apply-Lead” project, to design, implement, and evaluate 3 separate AI-focused learning journeys particularly targeting the aforementioned archetypes. 3 THE LEARNING JOURNEYS 3.1 Content In the summer of 2020, a group of MIT faculty and staff, with an expertise on AI and STEM education, collaborated with JAIC to develop the first set of learning journeys. Three cohorts were formed including learners representing two archetypes each, who appeared to have similar working needs, interests and backgrounds, and would equally benefit by attending the same courses. First cohort includes learners from the Lead AI and Drive AI archetypes (L/D) traditionally focusing more on management and leadership of the organisation, second includes learners from the Create AI and Embed AI (C/E) being mostly technology developers and facilitators, and third includes learners from Facilitate AI and Employ AI (F/E) who are mainly AI technology end users. After examining the needs of each cohort, the MIT team proceeded to explore what appropriate AI related courses the group could leverage on from MIT’s online catalogue of resources, along with what would be the most meaningful development of custom-made exploratory courses and accompanying educational experiences. Discussions led to the design of 3 different learning journeys employing a variety of pre-existing, adapted, and new educational resources. 3.2 Pedagogy and Educational Technology Educational research has already shown that online courses where students can access educational resources at their own pace and master topics and concepts before moving on to more advanced levels is an effective way to learn [5]. Hands-on authentic project-based and problem-based learning is another established way to master technical skills and applied practices [6]. However, existing online learning platforms often fall very short when needing to support automated personalised tutoring or collaborative hands-on projects. Considering the program scope and 50th Annual Conference in September 2022 1533 technological restrictions, the team decided to employ, study, and evaluate a combination of technology-enabled pedagogies and experiences, while combining online and in-person learning contexts for AI [7]. Furthermore, state of the art teaching and learning practices and theoretical frameworks have been employed in several aspects of each learning journey, such as active learning, memory retrieval practices, employment of low-stake quizzes, and presentation of worked/faded examples. Number of learners is also expected to significantly scale over time. Given the opportunity to leverage existing MIT resources while also developing new content, the group ended up offering 3 learning journeys that combined 5 learning modalities, that are now getting separately studied and evaluated (via 4 learner surveys spaced throughout the program, interviews to program staff, and “gap analysis” performed by MIT-USAF AI experts) to guide our understanding about optimum AI teaching approaches: ● online asynchronous self-paced courses about AI fundamentals were available to learners throughout the duration of the whole program ● online asynchronous instructor-paced courses (about 3-months in duration, each) ● experimental courses (online or in-person project and team-based courses, particularly designed to interweave technical and ethical content about AI, with a 3-5 days duration) ● live online events about AI offered by MIT to a greater audience (1-2 hours long webinars) ● curated supplementary materials available to learners to read or watch (spanning 30 hours of material) Table 2. Learning modalities employed in the 3 learning journeys Cohort Learning Journey courses and activities (# of offerings) Online asynchronous self-paced Online asynchronous instructor-paced Experimental Courses Online Experimental Courses In-person Live Online Events L/D (N=100) 2 1 (select out of 2 options) 1* 1* 6 C/E (N=30) 2 1-3 (select out of 3 options) N/A 1 6 F/E (N=100) 2 N/A N/A N/A 6 *learners could choose between one online or one in-person experimental course To better support the program, and given that the learners had to enrol in different platforms to participate in various pre-existing online courses, the team proceeded to develop an access portal for learners. Through this portal the learners were directed 50th Annual Conference in September 2022 1534 to their own personalised learning path and could keep track of their progress, while the development team could also keep track of learner’s progress and course completion status (using a self-report tool). 3.3 Learning journey content and timeline Figures 1-4 present the 3 learning journeys along with the planned timelines showing various offerings to each cohort. The learning journey for the L/D cohort was offered to 100 learners and designed with an emphasis on AI business strategy, ethics and policy. The learning journey for the C/E cohort included 30 learners and had a greater orientation towards development of deep technical knowledge and the knowhow of AI tech solutions with an emphasis on probability, statistics, and machine learning. The learning journey of the F/E cohort aimed to provide fundamental knowledge about AI to an end user of AI solutions. This cohort included 100 learners. Fig. 1. Lead/Drive Learning Journey with an emphasis on AI business strategy, ethics, and policy Fig. 2. Create/Embed Learning Journey with an emphasis on technical knowledge and know-how of AI solutions 50th Annual Conference in September 2022 1535 Fig. 3. Facilitate/Employ Learning Journey with an emphasis on developing fundamental knowledge about AI Fig. 4. Screenshot of the 3 learning journeys as presented to each cohort in our AI Education Research Portal 4 CURRENT STATUS AND FUTURE ITERATIONS 4.1 Program completion and certification April 15th, 2022 was the completion date for the first 3 cohorts participating in the program. Some activities were delayed due to COVID restrictions being imposed to in-person courses/workshops. Figure 5 presents the completion rate per cohort. Learners receive a certificate of completion if they have completed over 90% or the required tasks, videos, readings and low-stake quizzes (knowledge check-ins) of their learning journey (based on their responses in the AI platform self-report tool). Our team noticed that there were many optional tasks that had also been completed. 50th Annual Conference in September 2022 1536 Fig. 5. Percentage of completion of all required tasks as of March 15, 2022. 4.2 Future Plans Based on an initial first pilot evaluation study, described in great detail in [7], and also guided by recent findings from the Science of Learning field, plan for future improvement suggest work on the following areas: ● Technology: Improvement of accessibility and developing a program that will support single access to all courses. ● Content: Develop case studies and examples directly related to the learner organisation to provide better context and raise interest and engagement. ● Pedagogy: Redesign knowledge check-ins used in the asynchronous selfpaced courses, to better support retrieval learning practices and enhance learning retention. Provide the F/E cohort with opportunities for live mentored sessions. Provide and better support community building and opportunities for peer-learning for all cohorts. Consider more opportunities for cross-cohort interactions, as well as interactions with learners from other military units. Furthermore, a small group of the program staff will be interviewed to understand the logistical hurdles experienced by learners, and a group of MIT AI experts have been commissioned to perform a gap analysis on current curriculum and provide an additional set or recommendations about additional content and appropriate pedagogy. Overall, research findings will further inform updates to the future learning journeys. Scope of this program is not just to evaluate various learning modalities but also to understand if and how they would best serve the program as the number of learners will keep on scaling. The group is now working on the aforementioned improvements and has also proposed two larger cohorts to start within the next year. 5 ACKNOWLEDGMENTS Research was sponsored by the United States Air Force Research Laboratory and the United States Air Force [anonymous] and was accomplished under Cooperative Agreement Number [anonymous]. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorised to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. 50th Annual Conference in September 2022 1537 REFERENCE [1] Garrett, N., Beard, N., Fiesler, C. (2020). More Than" If Time Allows". The Role of Ethics in AI Education. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. pp. 272-278. [2] Nicastro, D. (2019). How Are Universities Responding to the Tech Skills Gap? CMS Wire2. Available on April 2022 from https://www.cmswire.com/digitalworkplace/how-are-universities-responding-to-the-tech-skills-gap/ [3] US Department of Defense (USDoD). 2019. Summary of the 2018 Department of Defense Artificial Intelligence Strategy: Harnessing AI to Advance our Security and Prosperity. Available on April 2022 from https://media.defense.gov/2019/Feb/12/2002088963/-1/-1/1/SUMMARY-OFDOD-AI-STRATEGY.PDF [4] US Department of Defense (USDoD), US DoD Joint AI Center (JAIC), DoD Chief Information Officer. Department of Defense Artificial Intelligence Education Strategy. 2020. Available on April from https://www.ai.mil/docs/2020_DoD_AI_Training_and_Education_Strategy_an d_Infographic_10_27_20.pdf [5] Avello M., Wiltrout, M. E., Bell, A., Haar, C. V., Fruchtman, S. (2020). Impact of Course Delivery Mode on Learner Engagement in MOOCs. Proceedings 2020 IEEE Learning With MOOCS (LWMOOCS). pp. 68-72, doi:10.1109/LWMOOCS50143.2020.9234325 [6] Sarma, S., Bagiati, A. (2020) Current Innovations in STEM Education and Equity Needs for the Future. Paper commissioned by the National Academies of Science, Engineering, and Medicine. Presented at the Imagining the Future of Undergraduate STEM Education Symposium on Nov 13, 2020. [7] Bagiati, A., Salazar-Gómez, A., Kennedy, K., D., Minicucci, N., Du, X., Breazeal, C. (2022) Designing and implementing an AI education program for learners with diverse background at scale. Accepted to be presented at the Frontiers in Education (FIE) 2022 Conference, in Uppsala Sweden on October 8-11.