Project completed, 2021–2022
All WorldSkills projects →The system a demonstration exam runs in, from checking in candidates to signed-off marks. Experts mark from a computer or a phone, the chief expert sees disagreements and the history of every mark right away, and the system produces the marking forms, reports and skills passports itself. One system for any number of exams instead of a separate copy of the software for each.
WorldSkills Russia runs demonstration exams: college graduates complete a task to WorldSkills standards, and a group of experts assesses the work against a marking scheme of modules, sub-criteria and aspects, both measurement and judgement. Marks were kept in CIS, the WorldSkills International Java application. From 2019 we ran it in the cloud: each exam got its own copy of the software with its own database and backups, and the more exams ran at once, the more copies had to be kept up.
The task was a system of our own, CIS NG 2.0: exams follow the same procedure as before, but in one system for any number of them. Plus what the old CIS lacked: sign-in with a code, expert permissions based on their role in a specific exam, change history and an action log, notifications, work from a phone, and an API through which external systems create exams and collect results.
A specification, a prototype and a design, then a web app for experts, a back end with an API, API documentation and a print-forms service. The app screens below show a made-up exam in the “Web Technologies” skill: the people and colleges are fictional.
A functional specification described every screen and its behaviour for four roles: chief expert, expert, observer and administrator. From it we built a clickable prototype of 37 pages and 17 dialogs, then the design: 114 layouts for desktop and for 320 px phones, and a kit of interface elements in WorldSkills Russia colours.
Screens and exam rules were agreed on the prototype before development, not on finished code.
“In focus”, “Unfinished”, “All” and “Bookmarks” tabs, filters by status, skill and dates, search. Each row shows the number, start time in local time, skill, venue, exam day, attendance and status: planned, attendance check, mark entry, mark lock, marks signed, completed. The chief expert starts and closes an exam right from the list.
An expert running several exams sees what is happening where and doesn't have to look for the right one.
The skill and the version of the assessment kit, the task variant, days C-1 and C1 in local time, the exam centre and its contacts, the chief expert, the expert group and study groups. The exam arrives from an external system through the API together with candidates and experts; the marking scheme and the task variant come from ESAT. The settings set the marking method: experts enter marks, or the chief expert does with confirmation by the experts; marks can only be unlocked through an administrator.
Everything about the exam on one page, without emailing the organisers.
For each day: present, absent or left early, with a reason, a comment and files, next to the face match percentage from the proctoring system. Moving a candidate to another team also needs a reason. For the first 15 minutes an attendance record can be corrected without explanation, after that only with a reason.
It's clear at once who came to the exam and whether it is the right person, and every correction is explained.
Modules, sub-criteria and aspects: measurement aspects carry a requirement or nominal size, judgement aspects a 0 to 3 scale with a description for each score. Every aspect has a maximum mark and a section of the WSSS skill standard.
Experts mark against the approved scheme, nobody copies it over by hand.
Experts enter marks online, and each one is saved at once. The chief expert sees every expert's score on a judgement aspect and the result; if the scores differ by 2 points or more, the aspect and the candidate are flagged for attention. Finished groups of aspects are locked, and if the connection drops the system warns the expert and blocks input. The old way still works too: the chief expert enters marks from paper forms.
Expert disagreements show up while marking, not when the mark sheet is signed.
One aspect for all candidates in a row: the scale with a description of each score in view, attached files and a counter of how many candidates are already marked.
An expert judges one piece of work after another by the same yardstick, so marks are more consistent.
Each mark keeps all versions for every expert: who, when, from which score to which, the reason, a comment and files. The log records sign-ins, attendance, team changes, changes to marks and settings and task downloads, filtered by author, event type and time.
Any disputed score can be traced: you can see who gave it and why it was changed.
Six sets of forms: blank forms, forms with aspects, forms for handwritten marks, forms with aspects and marks, and sign-off sheets for the marking scheme and for entered marks. A candidate's skills passport shows scores by standard section and by module, with a QR code and the chief expert's signature. Everything is rendered to PDF on the server.
The paper side of the exam stays, but the system fills in the paperwork, not the expert.
17 reports for the chief expert: the marking scheme and the WSSS mark allocation, marked and still unmarked aspects, mark lock progress and times, missing marks, “Experts head to head” showing who marks above or below the group, skill and region comparison, official results. Results open on screen, as a PDF or as a spreadsheet.
The chief expert sees how the exam is going and each expert's marking habits without compiling data by hand.
The back end runs on PHP 8.1 under RoadRunner with PostgreSQL, the web app on Vue 3 and TypeScript, and PDFs are rendered from HTML templates by a separate service. Everything can be managed through a documented API: external systems use service accounts to create exams with candidates and experts and receive results by webhook once the exam is completed. Experts sign in with a one-time code sent to email or phone, and their permissions follow their role in a specific exam. Emails and text messages go through several gateways with an even load on each and delivery checks.
Scale: 4 repositories (web app, back end, API documentation and print-forms service), about 1,500 commits, about a hundred API methods, 9 kinds of emails to experts, from appointment to an exam to its closing and reopening.
Instead of a separate copy of the Java software with its own database for each exam, a shared system that uses less memory and scales with the load.
Expert disagreements are flagged at once, every mark has a history with reasons, and every action in the exam is logged.
Exams, candidates and experts arrive through the API, the marking scheme comes from ESAT, and results go back to the external system on their own.
Sign-in with a code, a personal exam link by email, phone-friendly screens, times shown in the expert's time zone.
Project completed. We worked on the system in 2021–2022, with the latest changes in March 2022. The WorldSkills Russia movement has since been renamed.
Tell us about it: we reply within one business day and suggest where to start.
Telegram: @spacetab_bot · Email: bip@spacetab.io
