Design projects units for BSc (Hons) Design Engineering students at Bournemouth University integrate and apply knowledge from a range of taught units together with self-directed learning and towards solving design problems. Recently, level 6 (FHEQ) project students have proposed and designed solutions that require AI vision-systems. These projects presented a problem for supervision, with limited, or no expertise in the technology or available equipment; students therefore treated these subsystems as a “black-box” exercise. To address these issues a set of technical requirements were compiled, a range of AI technology solutions were identified before selecting the Nvidia Jetson Nano. From the literature, a stream-lined practical program was developed to introduce the technology to level 5 and level 6 project students as part of their design education. This provided hands on experience through familiarization with the interface and the use of pretrained models before students re-trained networks with their own datasets. Level 5 students utilised the technology to develop a scratch detection machine for sorting damaged components. Level 6 students were provided with the opportunity to integrate the technology into projects where appropriate and two students did so; one developed a device to identify people trapped in buildings after an earthquake, the second developed a device for monitoring chili-plants when grown under polytunnels. Developing and delivering the introductory programme as a non-expert learning pathway has enhanced the student experience within design education, provided a simple workflow that students can utilise and build upon, and led to successful student outcomes

Garland, Nigel

Palmer, Sarah

Wade, Russell

Bournemouth University Research Online

EPDE2023/1105 25TH INTERNATIONAL CONFERENCE ON ENGINEERING AND PRODUCT DESIGN EDUCATION 7-8 SEPTEMBER 2023, ELISAVA UNIVERSITY SCHOOL OF DESIGN AND ENGINEERING, BARCELONA, SPAIN NON-EXPERT PRACTICAL APPLICATION OF AI VISION SYSTEMS IN DESIGN ENGINEERING PROJECTS Nigel Garland, Russell Wade and Sarah Palmer Bournemouth University ABSTRACT (250 WORDS MAX) Design projects units for BSc (Hons) Design Engineering students at Bournemouth University integrate and apply knowledge from a range of taught units together with self-directed learning and towards solving design problems. Recently, level 6 (FHEQ) project students have proposed and designed solutions that require AI vision-systems. These projects presented a problem for supervision, with limited, or no expertise in the technology or available equipment; students therefore treated these subsystems as a “black-box” exercise. To address these issues a set of technical requirements were compiled, a range of AI technology solutions were identified before selecting the Nvidia Jetson Nano. From the literature, a stream-lined practical program was developed to introduce the technology to level 5 and level 6 project students as part of their design education. This provided hands on experience through familiarization with the interface and the use of pretrained models before students re-trained networks with their own datasets. Level 5 students utilised the technology to develop a scratch detection machine for sorting damaged components. Level 6 students were provided with the opportunity to integrate the technology into projects where appropriate and two students did so; one developed a device to identify people trapped in buildings after an earthquake, the second developed a device for monitoring chili-plants when grown under polytunnels. Developing and delivering the introductory programme as a non-expert learning pathway has enhanced the student experience within design education, provided a simple work-flow that students can utilise and build upon, and led to successful student outcomes. Keywords: AI, Projects, Machine Learning 1  INTRODUCTION Design projects are an essential element of design education learning for BSc (Hons) Design Engineering students at Bournemouth University and represent 20 ECTS credits at level 5 (FHEQ) and 30 ECTS credits at level 6. These projects allow students to integrate and apply knowledge from a range of taught units together with self-directed learning and towards measurable outcomes. Through these units, students demonstrate specific elements of the Engineering Council (ECUK) AHEP3 and AHEP4 learning outcomes for accredited degree programs [1]. For level 5 projects, students design solutions to both individual and group projects meeting the ECUK specification of “broadly-defined problem”. Level 6 students work on a single individual design project from their own proposal meeting the ECUK descriptor of “complex-problem” and require technical challenge. In either case, students are expected to develop new knowledge in subjects they may be unfamiliar with through identifying their individual learning needs and applying self-directed learning, preparing them for lifelong learning. 2 EMBEDDING AI TECHNOLOGIES For AY2020-21, a number of Level 6 students proposed final year projects with autonomous control systems, typically navigation or decision-making, requiring artificial intelligence (AI) vision-systems integrated to control system. These projects presented a problem for supervision, especially during the recent pandemic, as students had received no prior learning in the technologies, and the department had limited expertise or experience of AI technology. Students therefore treated these subsystems as a “black-box” [2] exercise within their project, rather than developing hands-on experience of integrating AI technologies and enhancing their design education experience. EPDE2023/1105 The need to develop capacity for AI learning within the department aligns with findings from the literature. Miranda et al [3] found AI/ML to be a core component of technology and competency for meeting Education 4.0. Laupchler [4] identified the importance in improving AI literacy of non-experts in higher education. Nakhle [5] identified the importance of AI in phenomic image analysis and the dependence upon data scientists, and present an interactive tutorial to aid researchers without coding experience. Essentially, AI literacy is an essential skillset in engineering design education. 2.1  Review of student project requirements A number of Level-6 students from 2020/21 designed projects embedding AI systems as a “black-box” within the control-system. In each case students provided an outline specification of expected outcomes (function) and specified a suitable technology (means); all students lacked detailed knowledge in the subject area to specify how the technology might satisfy those outcomes.  Level-6 final year projects embedding AI systems as a “black-box” model were identified together with the black-box function and proposed technology; each project was subsequently ascribed a suitable means and minimum performance in frames/second (Table 1).  Table 1. Level-6 project function-means translation Project Black-Box Function Technology Means FPS Beach Cleaner Identification of beach waste Vision system Object detection 20 Paddock weeder Identification of poisonous weeds in a horse paddock Vision system Image classification 40 Pothole Detector Identification of road surface damage Vision system Object detection 40 Marine surface cleaner Identification of floating waste in marinas Vision system Object detection 10 Automated medical supply unit Navigation within a Hospital Vision system Semantic segmentation, integration to ROS 20 Values identified within the table are estimated and dependent upon image size, camera orientation and rate of image variance. For example, Asad et al [6] found that a Yolov5 neural network retrained as a pothole detector achieved an accuracy of 90% at 38.9FPS when tested against a video stream from a vehicle travelling at 65kmh (40mph).  2.2 Suitable AI ecosystem A study of available AI ecosystems was conducted in order to provide students access to suitable equipment, knowledge and learning opportunities. Key objectives were: availability, cost, extent of ecosystem, transferability of skills, range of vision process techniques, ease of system-training, track record, deployability. Although the current market is now well served, at the time of the initial investigation there were limited options available that could meet the objectives (Table 2). Table 2. AI system attributes System Advantages Disadvantages Google Coral USB accelerator Low-cost. Compatible with Linux or Raspbian.  Wide range of pre-trained AI models.  Trains off-board using Google Co-lab & Jupyter notebooks. Good performance. Wider ecosystem includes developer boards. Poor availability when required. No semantic-segmentation retraining option.  Limited range of models. Requires training off-device. Requires a mini-computer for edge operation. Intel Neural Compute stick Low-cost. Compatible with Windows, Linux or Raspbian.  Wide range of pre-trained AI models.  Trains off-board through open-vino using Intel based PC.  Good performance. Poor availability when required. Training workflow is non-intuitive. Requires training off-device. High level of technical knowledge required.  Requires a mini-computer for edge operation. EPDE2023/1105 Nvidia Jetson Nano Good availability. Low-cost.  Linux developer board with GPIO.  Extensive ecosystem, including learning materials.  Used extensively industry.  Wide range of pre-trained AI models.  Trains on-board & off-board.  Very good performance.  Simple to integrate into control systems using Python.  Availability limited throughout 2022.  Recommended off-board training requires Linux PC with 32GB Nvidia GPU or similar cloud instance. From the evaluation it was clear that the Nvidia system held significant advantages in workflow, capability and useability; crucially, they were available. Seven developer boards together with cases, PSUs, SD cards and fans were purchased for evaluation and curricula development. 2.3 Deep Neural Networks on the Jetson Nano The Jetson Nano provides two practical routes to deployment of a range of Deep Neural Networks (DNN): Jetson-Inference, and Deepstream SDK. Jetson-Inference (Hello AI World) An Nvidia package of DNNs and runtimes and vision libraries for inference on Jetson devices. This package provides ten image recognition models, ten object detection models, eleven semantic-segmentation models and three pose estimation models (Figure 1). The package includes Pytorch [7] for transfer learning (retraining) of image recognition and object-detection DNNs onboard the Jetson.  Figure 1. image recognition (L), and object detection (R) Deepstream SDK A streaming analytics toolkit capable of deploying multiple DNNs across multiple video feeds with features such as object tracking and other sensor data. Industry standard DNNs can be downloaded from the Nvidia model zoo, trained off-board on an Ampere GPU equipped Linux PC, or trained in a cloud instance such as Azure.  3 NON-EXPERT LEARNING OF PRACTICAL AI Familiarity was gained through Nvidia Deep Learning Institute training courses [8]; experience here was distilled to two generalized workflows and supporting materials for a range of student use-cases: 1. Deploying pre-trained networks for classification, object-detection, semantic-segmentation, or pose estimation. 2. Retraining existing networks for custom use-case and deploying for classification or object-detection. Although Deepstream was evaluated alongside jetson-inference, the procedure for deployment of custom models was found to be more complex than for jetson-inference. Therefore, both workflows were developed exclusively through the jetson-inference package. EPDE2023/1105 4 PRACTICAL EXPERIENCE FOR PROJECT STUDENTS All project students gained practical experience in small groups through hands on familiarization of the Jetson Nano and deployment of pretrained DNNs for object classification, object-detection, semantic-segmentation, and pose estimation. Following the introductory training, Level-5 students go on to develop a practical application through retraining an object-detection DNN described below. Level-6 students that required AI control within their project were provided further learning opportunities in transfer-learning and python script editing to assist with their specific use-case.  4.1 Application to Level-5 student group project For the level-5 group project, students from both AY2021/22 & 2022/23 cohorts were tasked to design and build a part sorting machine that detects un-scratched and scratched plate components (Figure 2), sorting them into separate output bins.   Figure 2. un-scratched plate (L), and scratched plate (R). The scratch and plate were detected utilising the Jetson Nano with overall system control provided by a Schneider Electric M221 PLC.  For the AI workflow (Figure 3), students collected and annotated (labelled) more than 200 images, using the Jetson camera-capture tool to create their own ground-truth in the pascal-voc format [9].   Figure 3. Jetson transfer-learning workflow Students used the ground-truth to retrain SSD-mobilenet-v2 [10] onboard the Jetson using Pytorch through 80 epochs (training loops) before exporting to an open neural network exchange (ONNX) data format [11]. The onnx file can then be called by a python script that optimizes to the Jetson Nano’s Tensor RT format before running as an object detector. The Nvidia provided script can be further customized to extract key data from the detector (object class, confidence, co-ordinates) for specific actions as required, such as signalling to the PLC over GPIO. EPDE2023/1105 Level-5 outcomes All project groups from AY2021-22 and AY2022-23 successfully retrained the DNN with their own ground-truth. However, 2021-22 students initially struggled to improve their model performance during training due to human error in mislabelling of detection classes. Students were able to correct these errors by utilisation of third-party labelling tool CVAT. The experience in label errors led to revising the student learning for 2022-23; here the initial ground-truth was treated as a familiarization exercise to emphasize the care required in the labelling stage hence DNN retraining was more consistent. One group from each AY successfully demonstrated a fully integrated system (Figure 4).  Figure 4. typical student plate detection sorting machines The most common problems were related to the plate loading mechanisms, not the AI implementation.  Where the DNN did have performance problems it was typically misclassifying the background, when no plate was present, as containing a plate. Again, this was due to poor control of the training environment; typically, a lack of variation in backgrounds used during the collection of ground-truth images. Students had not constructed the hardware before training their DNNs, therefore training images could not be captured directly from the sorting machine. Where students varied the background, with random objects in-shot during capture of ground-truth, they had fewer misclassifications of detection. 4.2 Application to Level-6 student final projects Two students used AI tools within their 2021-22 final year projects. Both students developed their applications using the Jetson Nano after completing the familiarization training outlined above. Chili monitor The first student designed an autonomous device to monitor chili pepper plants and identify when the fruit was ripe. They followed a similar procedure to the level-5 group project to develop their technical demonstrator and identified the limitations of the DNN pipeline; it was not possible to count the chilis, but the student did identify how the same methods could be used with Deepstream to achieve this.   Figure 5. Chili monitoring robot, detection (L), design (R) The demonstrator operated successfully after retraining the mobilenetSSDv2 DNN with a limited number of images to detect ripe and unripe chilis. Similar to the group projects of 2021-22, the initial re-training was hampered by mis-labelling and limited quality ground-truth. Attempts to correct this were hampered after CVAT became unavailable with the withdrawal of Intel from their Russian EPDE2023/1105 operations but overcome with additional images from fresh plants and data checking using another 3rd party annotation provider, Roboflow. Earthquake building search tool The second level-6 student developed a tool for locating victims trapped in earthquake damaged buildings. In this case, the object being identified was a human, therefore a standard pre-trained network would be suitable for the task. The student experimented with a range of both object detection networks and semantic segmentation networks before selecting the multi-human-parser (MHP) segmentation network [12]. This utilised the jetson-inference pipeline with the original python code edited to provide GPIO output from threshold values for particular classes as a percentage of screen pixels. For the MHP network this was used to signal the presence of classes such as arm, leg, hair, face, body. The final design featured a series of lamps for body-parts, a screen for visual verification and telescopic camera manipulator to reach into enclosed spaces. 5 FINDINGS The introduction of AI tools through non-expert experiential learning has been successful. It has enable a pathway to deploying advanced technology in design engineering projects without the need for technical support from data scientists and only a basic knowledge of coding. Both Level 6 students benefitted from developing their knowledge on the same learning journey as their supervisors, a shared experience. Level 5 students were able to explore the possibilities of the technology without risk or detriment by learning through the shared experience of a group project where overall technical success or failure was not a significant part of the marking criteria. However, there were three principal weaknesses in the project experience:  First, the lack of verification tools (or complexity of) for pre-checking and correcting object-detection ground-truth on the device.  Second, level 5 students failed to adequately plan their work, underestimating time for building and testing of hardware elements.  Third, difficulty in obtaining hardware over the period 2021-23, although this appears to be in line with availability of many micro-processors and micro-controllers. REFERENCES [1] Engineering Council UK. The Accreditation of Higher Education Programmes (AHEP). in Fourth edition, ed. London: Engineering Council, 2020. [2] Cross N. Engineering design methods : strategies for product design. 2nd ed. ed., 1994 (Wiley,Chichester). [3] Miranda J., et al. The core components of education 4.0 in higher education: Three case studies in engineering education. Computers & Electrical Engineering, 2021,vol. 93, pp. 107278. [4] Laupichler M. C., et al. Artificial intelligence literacy in higher and adult education: A scoping literature review. Computers and Education: Artificial Intelligence, 2022,vol. 3, pp. 100101. [5] Nakhle F. and A. L. Harfouche. Ready, Steady, Go AI: A practical tutorial on fundamentals of artificial intelligence and its applications in phenomics image analysis. Patterns, 2021,vol. 2, pp. 100323. [6] Asad M. H., et al. Pothole Detection Using Deep Learning: A Real-Time and AI-on-the-Edge Perspective. Advances in Civil Engineering, 2022,vol. 2022, pp. 9221211. [7] Paszke A., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 2019,vol. 32, pp. [8] Nvidia. (2020, 06/03/2023). NVIDIA Deep Learning Institute. Available: https://www.nvidia.com/en-gb/training/ [9] Everingham M., et al. The pascal visual object classes (voc) challenge. International journal of computer vision, 2009,vol. 88, pp. 303-308. [10] Sandler M., et al. Mobilenetv2: Inverted residuals and linear bottlenecks. in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510-4520. [11] Junjie Bai, et al. (2019, 06/03/2023). Onnx: Open neural network exchange. Available: https://github.com/onnx/onnx [12] Li J., et al. Multiple-human parsing in the wild. arXiv preprint arXiv:1705.07206, 2017,vol. pp.  

Non-expert practice application of an AI vision systems in design engineering projects

https://eprints.bournemouth.ac.uk/39679/1/EPDE23%20Final%20Paper%201105-%20nigel%20garland.pdf

Non-expert practice application of an AI vision systems in design engineering projects

Abstract

Similar works

Full text

Available Versions

Bournemouth University Research Online