1. EDIT
  2. 1. Overview
  3. 1.1 Introduction to A+

Introduction to A+

A+ learning management system (LMS) enables teachers to variedly utilize both (1) automatic assessment and (2) interactive, smart learning content. First, A+ has no limitations on the implementation of automatic graders: you may create any kind of assignment in A+ as long as you are able to develop a grader program (often called just a grader) for it. Graders receive the student's submission as input and output its score and textual feedback. The feedback may be as simple or rich as you want and are able to programmatically generate. Second, A+ allows you to freely develop the online learning materials: you can control the HTML markup of the course web pages as well as the JavaScript code and CSS styles. For example, you may develop interactive visualizations in JavaScript and easily embed them into the course materials.

A+ is a complementary system to Moodle, the official LMS provided by Aalto Learning Services (LES) as MyCourses. You could think the difference between the two like MS Word (Moodle) and LaTeX (A+) - at least if we forget about VBScript for now. Moodle gives you a GUI with low threshold adoption, a wide range of tools for basic needs in teaching (sharing course material and assignments, collecting student submissions, quizzes, peer feedback and discussions) and, at Aalto, a number of licensed or free add-ins, like Panopto for creating and sharing videos, Stack for mathematical assignments, CodeRunner for programming assignments, and H5P for creating interactive contents. Compared to Moodle, A+ provides you with additional features for producing course material including assignments with automatic assessment for Computer Science needs, with the drawback of a high threshold learning curve. It is also possible to use the automatic graders with limited functionality via a Moodle plugin called Astra.

A+ has been used to automatically grade various kinds of assignments, for example:

  • Programming in Python, Scala, C, C++, JavaScript and MATLAB to name a few. Graders are often implemented either as a set of unit tests or as input/output tests that check the strings printed by the student's program.

  • Relational databases: SQL queries and relational algebra expressions.

  • Web frontend (JavaScript, CSS) and backend programming.

  • Theoretical computer science: finite-state automata, regular expressions, context-free grammars and parse trees.

  • Mobile applications on Android.

  • Creating containers with Dockerfiles.

  • Interactive exercises on the Acos server platform, for example: point-and-click, drag-and-drop, Jsvee program animations, and Parsons programming puzzles.

  • Questionnaires with multiple-choice questions and text entry questions. Text entries may be automatically graded with regular expressions.

  • It is also possible to manually assesses assignments in A+. There is also an A+-compatible tool called Rubyric that focuses on manual grading with rubrics.

In A+, course contents and assignments are developed in open, plain text file formats. You own the materials yourself: there is no vendor lock-in in proprietary file formats or WYSIWYG editors that can not export the contents to an open format. You can freely develop and edit the materials in your own computer using the editors and development tools of your choosing. Often-used file formats in A+ include HTML, RST and Markdown which are converted to HTML, as well as YAML and JSON for assignment configuration files. Automatic graders can be developed in any programming language. If need be, you could convert your course materials to any other format and import them to another learning platform. Furthermore, the static web pages in the HTML format (excluding automatically assessed assignments) could be easily deployed to any web server without relying on any LMS software.

A+ course contents are stored in a repository under Git version control. Git keeps track of the changes in your course and you can also update the course in the server by simply pushing to the Git repository.

While developing course materials and assignments, you can run the whole A+ platform on your own computer in Docker containers. You can test new materials without interfering with the server while real students are accessing the course.

Automatic graders

In order to grade an assignment automatically, you must first develop a grader program for it. Automatic graders are usually developed on the MOOC-Grader platform. MOOC-Grader has two main features for grading:

  1. questionnaires that are graded synchronously and

  2. container-based asynchronous graders that you program yourself.

MOOC-Grader is an exercise service (also known as grader service) for A+. It is also possible to develop new, custom exercise services, but usually that is not necessary.

In the MOOC-Grader, graders for assignments are stateless. The grader

  • grades one submission at a time,

  • receives the student's submission as input,

  • outputs the score (a whole number) and feedback for the submission,

  • consists of any files and programs that the course staff has created using the technologies of their choosing,

  • is run in a container that securely isolates the execution from the platform and other submissions, and

  • has no access to data about the student's other submissions or other students' submissions.

The student's submission may consist of uploaded files, data inserted into a form or by some other interactions, or a git repository which the student has pushed the solution into. The score or points of the submission are given as a whole number out of the maximum that may be set freely. The grader may use a different maximum score than the A+ platform, in which case A+ scales the points to the maximum in A+. The feedback generated by the grader may be formatted in plain text or HTML markup. Typically, automatically graded assignments show the feedback to students as soon as it is available, but the feedback may also be delayed so that the students gain access to it only at the time specified by the teacher, for example, after the deadline.

Grading containers are normally based on Debian Linux and built with Docker. You may install any necessary tools, frameworks and libraries in the Docker image of the grading container. We have several Docker images available that have basic tools for different programming languages installed. You may also define your own Dockerfile and use that image for grading. We recommend that grading containers are based on our grading-base image that includes some utilities and configurations for compatibility with the MOOC-Grader.

A new container is launched for grading each submission. The MOOC-Grader server may grade multiple submissions concurrently, thus multiple containers could be running at the same time. It is also possible to configure a grading server so that it runs only one submission at a time. This is useful if the grader needs to measure the execution time of the submission. Measuring time is more reliable when the hardware is not contested by multiple processes.

Often, the grader in the container starts with a Bash script that is sometimes named "run.sh". Run.sh typically manages preparations for the grader and then starts it. For example, in a Scala programming course, run.sh could first compile the submitted Scala code, set the CLASSPATH environment variable so that the process finds the necessary Scala libraries and finally run the unit tests that output the points and feedback. After the process has finished, the container sends the results back to the MOOC-Grader, which then forwards them to A+.

Course study materials

One A+ course consists of modules (also known as exercise rounds, rounds or weeks depending on the course). A module has an opening time and a closing time (deadline) that restrict assignment submissions. It is also possible to open the study materials of the module before the assignments (by setting the "read opening time"). A module may be set to allow late submissions until the late submission deadline. A late penalty that deducts a percentage of the score may be applied to late submissions. The teacher may grant personal deadline extensions ("deviations") to students.

A module consists of chapters and assignments. Chapters form the study materials, which could contain, for example, text, images, embedded videos, specialized visualizations, and of course automatically graded assignments embedded in the chapter. You may freely use web technologies (JavaScript, HTML, CSS) in order to develop specialized tools when necessary. If you only want to write text, it is easy to do so in RST. It is also possible to include only assignments without any chapters in the course.

A+ chapters are often written in the RST markup, but using RST is not mandatory. Sphinx, the RST compiler, can also compile Markdown files (using recommonmark or MyST). Sphinx is a tool for creating documentation that can be compiled into multiple formats, such as HTML and LaTex PDF. Sphinx itself extends the Docutils RST parser and compiler.

For courses using the RST content format and/or Sphinx, A+ includes a module called A-plus-rst-tools. A-plus-rst-tools comprise a set of Sphinx extensions that contain useful RST directives for A+ course materials, particularly the directives for embedding assignments in chapters. A-plus-rst-tools are included in the course repository as a Git submodule.

If you don't like writing RST or Markdown, you could also write HTML directly. A+ has only a couple of requirements for the structure of the HTML document so that it can be used as an A+ chapter. The requirements are specified in the CONTENT.md documentation. You could also write chapters in any other format that can be compiled into HTML.

Architecture of A+

The main components of A+ are the frontend server and exercise services. The frontend server is responsible for

  • the student's uniform user interface,

  • retrieving course materials and assignments from the backend Git manager service and exercise service,

  • forwarding submissions to the exercise service for grading,

  • storing submissions and grading results (points and feedback) in the database, and

  • teacher's functionalities such as inspecting submissions and manual assessment.

The figure below presents the architecture of the components. The figure includes only one exercise service, the MOOC-Grader.

The teacher edits the course contents on his/her computer and pushes the changes to the Git server. The Git repositories are typically hosted on a GitLab server. (At Aalto University, the GitLab server is called Version.) The GitLab project is configured with a webhook so that it notifies the A+ Git manager service of the course update. The A+ Git manager service pulls the update and builds the course. Building includes, for example, compiling the study materials written in RST to HTML. You define yourself what processes are run during the build. For example, you could compile materials written in other markup than RST, or package source code templates (skeleton code) into archives that the students download as a starting point for a programming assignment. (If you package files in the build, you don't need to manually package them and store the archive in the Git repository.)

The A+ frontend retrieves the contents of the assignment from the exercise service. The exercise service to use is defined by the assignment settings. The retrieved assignment content is shown to the student in the A+ website, and it typically includes the instructions for the assignment and a form for making the submission (e.g., uploading a file). For an interactive assignment, the content could include a specialized editor that the students use to create their solutions. You may include any JavaScript code in the assignment in order to implement specialized editors or widgets. When the student submits, A+ saves the submission in its database and sends the submission to the exercise service for grading. The exercise service sends the points and feedback back to A+ (not necessarily immediately). In the case of the MOOC-Grader, when it receives a new submission from the A+ frontend, it launches a new grading container. The container sends the grading results back to the MOOC-Grader, which sends them to the A+ frontend.

The most used exercise service is the MOOC-Grader platform. Other widely used services include Acos server (demo) and Rubyric. Acos server is a platform for distributing browser-based smart learning content in a reusable and interoperable way. Rubyric is a tool for manual assessment: it supports pre-defined grading rubrics that may be used to score submissions and to provide feedback. The feedback may also be freely modified so that it is not constrained to the rubric. In addition, Rubyric has limited support for peer reviews between students. It is possible to develop new, specialized exercise services, but usually it is not necessary. The A+ frontend connects to the exercise service with the grader protocol, which uses HTTP GET and POST requests with a few parameters. The protocol is described in the GRADERS.md documentation. A new exercise service could, for example, be stateful as opposed to the stateless MOOC-Grader. It could combine data from multiple submissions and retrieve additional data from the A+ REST API. Its implementation does not have to depend on containers like the MOOC-Grader does.

It is worth mentioning the Radar service. It is not an exercise service, but a tool for the similarity analysis (plagiarism detection) of the submitted source code in programming assignments. Radar retrieves submission data from A+ via the API.

Note

We have said that the MOOC-Grader launches containers for grading submissions. When you run A+ on your own computer during the course development and testing, the containers are run on Docker. However, in the production servers at Aalto University, the containers are run on Kubernetes. This makes no practical difference in most courses and the grading containers function locally in the same way as in the production servers.

Student           ________________
  |              | Database:      |
  | http         | submissions,   |   Course staff/Teacher
  |              | grading results|                  |
A-PLUS-FRONT --->|                |  Push the course |
  |              |________________|           to Git |
  | http                                             |
  | Fetch exercise                                   |
  | Grade submission          Build the course:  ____v__
  |                                   make html |       |
MOOC-GRADER <-----------------------------------|  Git  |
    |                          a-plus-rst-tools |_______|
    |-- _build/yaml/index.yaml                   |
         |                                       |-- index.rst
         |-- _build/yaml/asgn_hello_python.yaml  |    |
                  |                              |    |-- chapter1.rst
                  | Grade submission             |
                  | Docker/Kubernetes            |-- exercises/
                  |                                  |-- hello_python/
      apluslms/grade-python:3.9-4.3-4.0                  |-- grader_tests.py

Structure of the Manual

This A+ Manual (or Aplus Manual) has been created as an A+ course.

This first module of the manual course provides an overview of the platform. The rest of the modules explain topics in more detail.

Posting submission...