Towards Standards-Compliant Assistive Technology Product Specifications via LLMs

The pre-processing step in CompliAT focuses on analyzing the product specification ($SPEC$) and the standard text ($STAN$), and preparing the text for analysis for the subsequent three standards compliance tasks. For instance, we employ preliminary NLP pre-processing steps [10], such as noise removal (i.e., stripping the two input documents of any extra or irrelevant information), tokenization, text chunking (to identify key noun phrases) and named-entity recognition (NER) for identifying the products or other categories of interest in $SPEC$ and $STAN$. For example, our Smart Knee example includes “StrideTech-ProKnee”, “StrideTech-ProKnee app”, “fall detector”, “smartphone”, “iPad” etc as NER entities. Note that pre-processing $STAN$ is a one-time event, and the results can be stored for any future analysis.

For the terminology consistency checking task, we focus on identifying and extracting the main keywords from both the input documents based on the noun phrases identified earlier, according to our previous work on requirements glossary extraction and clustering [11]. Once the keywords are identified, we use the NER results from the pre-processing step to select only the keywords of interest, such as product names or domain-specific keywords. After that, the keywords from $SPEC$ and $STAN$ are matched against each other to identify inconsistencies using string similarity matching (embeddings) and heuristics on the threshold for a match. The output of this task is the termnological consistency checking report for a given $SPEC$, which can be passed onto a human expert for review. For example, in our initial Smart Knee description, under ISO9999, it would be classified under orthotic and prosthetic devices related to lower limbs.

After task 3, $SPEC$ would need to be further checked for adherence to ISO 8549 and ISO 10328 standards, including load testing and vocabulary usage. These could include ISO 8549-1: General terms for external limb prostheses and external orthoses – use of definitions related to prosthetic devices, including what constitutes a knee prosthesis, the definition of a microprocessor-controlled prosthetic knee, and terms related to user safety and mobility; ISO 8549-2: Terms relating to external limb prostheses – definitions around support and stability, and context for the intended use and user needs that the StrideTech-ProKnee addresses.

We build a retrieval-augmented generation (RAG) approach [12] using large language models (LLMs) to implement this task. Product classification is a challenging task, as there could be hundreds of product categories in a given $STAN$, and it is difficult to build a classifier (e.g., a machine learning classifier) that can yield accurate results for multi-class classification with little training data and so many classes. For instance, StrideTech-ProKnee will be classified into more than one product category in ISO9999. We thus implement a hierarchical classification approach using RAG. Specifically, the process begins with the construction of a comprehensive knowledge base that includes detailed information from $STAN$, such as categories, definitions, and examples of previously classified products. This knowledge base is crucial for the retrieval component of the RAG model, enabling it to pull relevant information for product classification. The technical approach for hierarchical RAG-based classification and knowledge base will be additional key contributions of our research.

For each new $SPEC$, the RAG model initiates a retrieval phase, querying the knowledge base to fetch relevant information based on semantic similarity, focusing on context and meaning rather than exact terminology matches. Utilizing the retrieved information, the generative component of LLMs predicts the product’s classification, applying a hierarchical process to pinpoint the precise category by navigating from broader classifications down to specific subcategories. This helps manage the complexity of multiple categories and ensures accuracy. For example, for StrideTech-ProKnee specification, the RAG model will first classify the product in class 06 Orthoses and Prostheses and subsequently into other sub-categories, e.g., 06 24 33 Knee units.

The third task focuses on (i) identifying other relevant standards that the $SPEC$ should comply with based on the product classification results, and (ii) generating compliance recommendations for the current $STAN$. Currently, there is no systematic way of knowing which standards an AT product should comply with for a given country. As a part of CompliAT development, we plan to develop a knowledge base to identify the standards relevant to a given product in a specific country. This is equivalent to the requirements traceability task for the actual compliance part, which has been widely studied in the RE literature. We build another RAG-based approach for tracing the key requirements in $SPEC$ to the $STAN$. The knowledge base in this task will greatly benefit all stakeholders in this research, as no such comprehensive databases exist.

In our StrideTech-ProKnee AT example, the RAG approach would generate trace links to the relevant testing processes and conditions in ISO 10328:2016, for each key product specification. This could include test methods for static, dynamic, and fatigue strengths, i.e., how the prosthetic knee withstands physical stresses and mimics natural gait patterns without failing; environmental testing – subsections relevant to how prostheses must perform in various environments, including the sensor’s role in monitoring the knee and surrounding environment in the StrideTech-ProKnee; and requirements for user safety and performance – parts of the standard that address safety testing, especially related to the prevention of falls and ensuring stability.

Thereafter, we can also use the generative components of LLMs to provide a preliminary analysis of product specifications compliance with standard processes. The compliance rules will be drafted in natural language and inserted as prompts to LLMs. The output of this step is a requirements compliance results report on the other relevant standards, the traceability to the current $SPEC$, and compliance analysis. Engineers and acceptance testers can use these traceability documents to carry out analyses that all relevant standards and tests are being met by the device.

Standards are essential for ensuring a baseline level of quality, enabling clear communication, evaluating safety, and facilitating the comparison of products [13, 14]. Many countries have standards for AT products, and some follow the International Organization for Standards (ISO) for compliance purposes [15, 16]. A number of studies have proposed better harmonisation of diverse AT standards [1]. Several others have emphasized the importance of three CompliAT tasks proposed in this paper [17, 18, 19]. For instance, Elsaesser et al. [19] emphasized that the lack of standard terminology in the AT domain leads to numerous communication issues and challenges for the globalization of AT products and services. Bauer et al. [16] develop an AT device classification using ISO 9999, aiming to be consistent with various legislative acts. None of the previous works in AT research have leveraged advanced NLP techniques and attempted to build comprehensive knowledge bases to facilitate standards compliance.

Requirements compliance to regulations has been extensively studied in the RE literature due to the increasing complexity and rigor of regulations across various industries such as finance, healthcare, and assistive technology [20, 21, 22]. Several studies in RE have introduced (semi-)automated approaches for identifying, interpreting, and integrating regulatory requirements into the software development lifecycle or checking their compliance against requirements [22], with the compliance rules specified in various formats, e.g., natural language [23], templates [24], activity diagrams [25], formal notations [26], and semantic web-based methods [27]. The advent of NLP and ML technologies has opened new avenues for enhancing regulatory compliance in RE. Studies such as those by Cleland-Huang et al. [28] and Guo et al. [29] have showcased the potential of NLP techniques in automating the traceability of requirements to regulatory documents, thereby reducing the manual effort involved and improving the accuracy of compliance checks. Additionally, research on the application of other AI techniques such as ML, question answering [30] or generative AI for predicting compliance issues and guiding the requirements elicitation process reflects an ongoing shift towards more intelligent and adaptive compliance management systems [31]. These advancements underscore a growing recognition of the need for innovative approaches to navigate the complexities of regulatory compliance, ensuring that software systems not only meet functional requirements but also adhere strictly to the regulatory frameworks governing their operation and use. Having said that, to the best of our knowledge, none of the previous works in RE have addressed the issue of standards compliance and specifically for AT product specifications.