A Pilot Study for Analyzing Systems Engineer-Conversational GenAI Interaction: A Case Study for Requirements Development &amp; Validation

Emilie Perreau; Romain Pinquié; Cédric Masclet

doi:10.65351/hsi24.003

Abstract

With the democratization of Large Language Models, academics and professionals are searching for new use cases of conversational Generative Artificial Intelligence (GenAI) for systems engineering, including requirements engineering. This paper presents a pilot study to understand the impact of guidelines and templates on the interaction between ChatGPT and a systems engineer for developing system requirements. Results show that when appropriately used, prompting guidelines and templates improve the quality of requirements. Still, without domain knowledge, the GenAI cannot generate outputs with the quality expected by requirements engineering international standards.

View the INCOSE record

Download the paper

Citation

BibTeX citation:

@inproceedings{perreau2024,
  author = {Perreau, Emilie and Pinquié, Romain and Masclet, Cédric},
  editor = {Masson, Dimitri},
  publisher = {INCOSE},
  title = {A {Pilot} {Study} for {Analyzing} {Systems}
    {Engineer-Conversational} {GenAI} {Interaction:} {A} {Case} {Study}
    for {Requirements} {Development} \& {Validation}},
  booktitle = {Proceedings International Conference on Human Systems
    Integration},
  volume = {3},
  date = {2024-08-29},
  url = {https://hsi-wg.github.io/conferences/HSI2024/proceedings/a-pilot-study-for-analyzing-systems-engineer-conversational-genai-interaction-a-case-study-for-requirements-development-validation.html},
  doi = {10.65351/hsi24.003},
  langid = {en},
  abstract = {With the democratization of Large Language Models,
    academics and professionals are searching for new use cases of
    conversational Generative Artificial Intelligence (GenAI) for
    systems engineering, including requirements engineering. This paper
    presents a pilot study to understand the impact of guidelines and
    templates on the interaction between ChatGPT and a systems engineer
    for developing system requirements. Results show that when
    appropriately used, prompting guidelines and templates improve the
    quality of requirements. Still, without domain knowledge, the GenAI
    cannot generate outputs with the quality expected by requirements
    engineering international standards.}
}

For attribution, please cite this work as:

Perreau, Emilie, Romain Pinquié, and Cédric Masclet. 2024. “A Pilot Study for Analyzing Systems Engineer-Conversational GenAI Interaction: A Case Study for Requirements Development & Validation.” In Proceedings International Conference on Human Systems Integration, edited by Dimitri Masson. Vol. 3. INCOSE. https://doi.org/10.65351/hsi24.003.