CROWDSOURCING IN REQUIREMENTS ENGINEERING AND QUALITY CONTROL SYSTEMS Jordan J. Clayton, University of Colorado, Colorado Springs

Abstract

In recent decades, the increasingly global market has facilitated an environment where the needs for increasing quality, decreasing cost, and increased speed of development is constantly and competitively being pushed towards the asymptotes of ability. With the now established network of instant communication on the internet, an existing method of problem solving known as crowdsourcing has the potential to revolutionize the way that systems are designed and problems are solved. This paper intends to analyze crowdsourcing and its implications within systems engineering.

?

Introduction

Crowdsourcing is the act of outsourcing tasks that an employee or contractor would otherwise perform, to a large group of individuals or groups often in un-related fields of expertise and background. Crowdsourcing is an umbrella term that is used to describe a variety of ways to utilize the wisdom of crowds to help solve a problem, answer a question, or to make improvements to a system. The advancements of communication technologies and networks in recent years have made it possible to harness crowdsourcing in ways unimaginable before. We now can call upon thousands or even millions of individuals across the globe and across virtually all fields of expertise to volunteer their input and collaborate on a single point of focus. Effective use of such capabilities could have industry changing impacts across many fields and system engineering processes.

?

This paper seeks to gain an understanding of why crowdsourcing applications will dominate many future applications of requirements engineering and quality control systems, while also understanding the limitations of crowdsourcing applications within systems engineering.

?

Background

The topic of this paper is crowdsourcing and its applications in requirements engineering and quality control systems within the domain of systems engineering. To accomplish this an analysis was performed primarily on three main papers. These works include the following: “Crowdsourcing: A Primer and its Implications for Systems Engineering” conducted by Dr. Richard J. Gerth from The National Automotive Center and Dr. Alex Burnap from the University of Michigan, who presented their paper at a symposium for ground vehicle systems engineering, “CrowdREquire: A Requirements Engineering Crowdsourcing Platform” written on the behalf of Masdar Institute of Science and Technology Abu Dabi, United Arab Emirates, “Quality control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions” conducted by Florian Daniel from Politecnico di Milano, Pavel Kucherbaev from Delft University of Technology and others. Several other papers and texts were reviewed and are included as citations in text.

???????????????????????????????

Crowdsourcing Elements

According to Gerth and Burnap (2012a) there is a general structure of crowdsourcing containing six elements. The first element is the Task Statement. This is a statement of what the crowd is expected to do. Other names for this are the problem statement or the challenge depending on the type of crowdsourcing. Crowdsourcing types will be discussed later in this paper. These statements can be short and simple to fully unified systems and everything in between. The second element of the system is the Task Submission. This is the desired output or what it is that the crowd is to produce. Third is the Open Period and is the time frame from when the task statement has been made available to the potential recruitment or volunteer pool to when the task is set to be complete or product to be delivered. In general, the open period should be long enough for the crowd to complete the task, yet significantly shorter than the time that would be allotted to for a traditional process.

?

The fourth element is the Reward. Rewards can range from a monetary prize to simply a recognition for task completion. The reward structure has a direct effect on the motivation for the crowd to respond, so this is important to understand. The Crowd itself is recognized as the fifth element in the system. It is defined as any group of people that can be a restricted set, such as only individuals with a Top-Secret clearance, or it can be open for anyone on the planet that can access the task statement. Some companies such as Innocentive.com, conduct challenges that require any individuals that participate to sign a legal document that give intellectual property (IP) rights over to the client prior to receiving the Task Statement.

?

Element 6 is Crowd Management, which refers to the processes and tools used for crowd recruitment, retention, involvement, fairness of participation, communication between participants and those responsible for management of the task. The crowd management is an explicitly controllable variable and can make or break the outcome of the outcome depending on the type and complexity of the task of interest.

?

Types of Crowdsourcing Tasks

Distributing simple tasks to individuals is not a particular challenging process, however, this is not the case with more complex tasks. Complex tasks require a much higher level of organization as noted by Adepetu (2012a). This is especially true when the solution contributes to a subtask that will then contribute to a comprehensive solution. There are five main types of crowdsourcing solutions that include grand challenges, evaluation, content repositories, outsourcing, and broadcast search.

?

Grand Challenges

Grand challenges typically have complicated problem statements which require demonstrating a prototype. The prototype is often evaluated based on performance criteria and these usually have a relatively long open period and much higher rewards than the other kinds of challenges that are only distributed to the winning individual or team. Because typically only the winner receives compensation, traditional corporations do not often invest in the amount of R&D funding that would be required for a complete prototype. This sets the stage for most of these challenge participants to come from organizations that have ample financial backing so that the business risk is low in the case they do not win the prize.

?

An example of a grand challenge is DARPA’s SubT challenge. This challenge has a goal to develop new technologies that can rapidly map, navigate, and search intricate underground environments such as tunnel systems, storm drainage systems, sewar systems etc. Teams can compete on a virtual course or a physical one in the Systems Competition. They are judged on autonomy, networking, perception, and the mobility that their system can demonstrate (darpa.mil). It is recommended that grand challenges only be used for when a proof of concept for an entire system is needed as they are not predisposed to optimizing subtasks or interfacing.

?

Evaluations

Evaluations are a more simplistic and a more easily applied type of crowdsourcing. One of the easiest examples of an evaluation is Amazon.com reviews. The crowd rates a product based on a 5-star scale and can leave comments justifying their score for an added layer of detailed analysis. Engineers are very often required to evaluate between alternatives for various factors, which makes evaluations one of the highest value-added tasks that crowdsourcing can be used for.?

?

Content Repositories

Content repositories are another form of crowdsourcing a system that have high value added to effort applied ratio. These can range from simple repositories such as YouTube.com where all the content creation is done by the crowd and simply uploaded to the internet through youtube.com. Wikipedia.com, on the other hand, uses collaborative crowd management to create content through organized teams.

?

The most significant impact of these kinds of crowd-based systems is in opensource software development. Github.com is one of the best examples of an opensource software development repository and one of the best examples of how crowdsourcing can be effectively leveraged to use within complex systems (Gerth, and Burnap, 2012b). Github.com provides the infostructure and tools that allow individuals in numbers of up to the tens of thousands to collaborate on a single data source. The way Github.com works, is an individual or group of individuals start a new content area that seeks to solve some problem or enhance an existing solution. Others who are also interested in this project can participate through a structured and layered process of contribution.

?

The different layers of production consist mainly of those who write code, those who debug the code, and those who improve on the basic code by “branching” the code to a separate subtask that may be merged back into the main project after its own iterations of code writing, debugging, and verification. Exhibit 1 displays the flow of the branching process that occurs on Github.com.

Exhibit 1. GitHub Project Flow Diagram (guides.github.com).

?The nature of crowdsourcing facilitates the development of ideas as a project is in progress. When new features are deemed feasible and value adding enough, they can be added to the primary project as a branch. The branch is an environment where these new features or ideas can be worked out independent of the primary project. Any changes that occur in a branch do not affect the primary project or primary branch. Once a branch has been established, there is an interface that keeps track of all additions, edits, deleted files, or comments, allowing for a transparent history of the work that has been done.

?

This thoroughly documented timeline allows for other individuals to join in and contribute intermittently or consistently. The next step in the process is the pull request. A pull request initiates discussion about the changes made in the branch. This allows for analysis and validation of the branch and a determination can be made as to whether it should be pursued further and if so, what changes need to be made if any. Once the pull request has been reviewed and the branch has passed preliminary testing, it may be deployed to have the changes verified in production testing.

?

There are different deployment strategies for different teams depending on their workflow preferences. Some teams have a specially provisioned testing environment in which deployment takes place, others deploy directly into the production. Once deployment testing and verification is complete, a merge into the main branch may be initiated. All the pull request comments remain preserved as a record of changes made to the code. These comments are searchable within the interface, which allows anyone who looks back into the history to understand when and why certain decisions were made (guides.github.com).

?

Outsourcing

This type of crowdsourcing task requires that a system be broken down or decomposed into small and well-defined units of work that are then outsourced to the lowest bidder. This is sort of an extension to standard contracting work but with the use of the internet to facilitate the contracting to much larger and diverse groups of individuals or teams. The advantages of having a much larger pool of potential contractors are both cost and quality of work. An example was given that explained a scenario where Java coders were hired to write script that described a basic repair flow for airline maintenance. Using a traditional dedicated team usually cost this company $2,000 per script, however, with the use of outsourcing to the crowd this cost went to $5 per script (Gerth, and Burnap, 2012c).

?

Broadcast Search

Broadcast search is the last type of crowdsourcing that will be discussed in this paper. Broadcast searching is related to outsourcing in that the task will be broken or decomposed into small well-defined units of work to be outsourced. The difference is that the decomposition and definition must be adequate for individuals rather than teams to be able to solve and submit solutions to the problems. The companies that provide the infrastructure to perform these types of crowdsourcing are web based. One such company is Innocentive.com. A study of Innocentive.com was conducted by Jeppesen and Lakhani (2010) from 2001 to 2004 that gathered some useful statistics on the problem-solving rates. They found that 30% of the problems that were added were solved and 30% of those had more than one viable solution submitted, 16% of the individuals signed up to be problem solvers read the problem statements, 7.8% of the 16% that read the problem statements submitted solutions and 6% of the submitters were winners. Another significant finding was that non-experts in the field submitted more winning solutions relative to the proportion of non-experts who submitted solutions. This finding suggests that diversity of experience is one of the most important characteristics of the crowd participating in the system.

?

Requirements engineering

Requirements engineering involves multiple stages which include elicitation, analysis, specification, and quality assurance. If these are organized properly, crowdsourcing may be utilized in a productive manner (Adepetu et al., 2012b). Exhibit 2 illustrates some top-level requirements that would be starting points for determining requirements of a system.

Exhibit 2. System Design considerations (Blanchard and Fabrycky 2011).

?Exhibit 3. Decomposing System Design Requirements (Blanchard and Fabrycky 2011).

Once the design requirements have been determined and prioritized from Exhibit 2, they may be further broken down into the subsystem level requirements as seen in Exhibit 3 (Blanchard and Fabrycky 2011a). Here the definition of the needs at the top system level must be properly defined before determining customer requirements and moving on to creating design standards. Describing the functions that must be performed is how the requirements for the system are founded. Operational and support functions must be described along with general concepts and inflexible requirements for production, integration, and retirement.

?

Decomposition of requirements

It is essential that the development of design criteria be based on an appropriate set of design considerations, considerations that lead to the identification of both design dependent and design independent parameters and that support the derivation of the technical performance measures. It is critical that the design criteria are based on appropriate design factors so that design dependent parameters (DDPs) and design independent parameters (DIPs) support the derivation of the technical performance measures (TPMs).

?

DDPs are attributes and/or characteristics that are intrinsic to the design. Some examples of DDPs would be design life, weight, reliability, producibility, and maintainability, etc. DIPs are factors that are external to the design and will likely have to be estimated or forecasted. Some examples of DIPs would be cost of fuel per lb. of weight, labor rates, material cost per pound, and future interest rates (Blanchard and Fabrycky 2011b). TPMs are predicted or estimated values for DDPs. These values are used as benchmarks to determine the performance of the system, therefore, should be considered very carefully.

?

Exhibit 4. CrowdREquire Use Case Diagram (Adepetu et al., 2012).

?CrowdREquire is an example of a company that provides a platform for utilizing crowdsourcing for complex systems. Exhibit 4 displays a high-level system flow diagram that shows the relationship between the client or actor that is defining the What that needs a How, the CrowdREquire platform, and the crowd that is providing the solution/solutions to the How question. For CrowdREquire to operate as a business, it was necessary that they understand their place within a system and the requirements to operate successfully. One of the requirements is that it needed to operate as a web-based platform. Web-based systems are best able to maximize many of the benefits of crowdsourcing like recruitment of large groups of people from all over the planet with a vast variety of experiences and expertise. Studies show that the best results come from groups with expertise in many domains rather than groups with expertise exclusive to the domain relating to the task statement. This is referred to the “wisdom of the crowds”, which assumes that no complete experts exist for sufficiently complicated problems and that the crowd on average is more likely to successfully solve a task than an individual.

?

Exhibit 5 displays a list of general requirements generated by CrowdREquire necessary for a platform to successfully use distributed internet-based problem solving or crowdsourcing. Requirements Specifications number 1 through 19 are DDPs and numbers 20 through 28 are DIPs.

?

Exhibit 5. CrowdREquire Requirements Specification (Adepetu et al., 2012).

Quality Control Systems in Crowdsourcing

The control of quality is actively researched and implemented in many disciplines currently. Some of these disciplines include software, data quality, user interfaces, and product build quality. Much can be learned from the lessons learned from these fields and transferred into quality control in crowdsourcing, however, it must be taken into consideration that the user or agents being crowdsourced are directly or indirectly involved in the task performance. Because of this, it is essential to understand the quality of output in crowdsourced systems is subjective to many factors that are not typically considered in many traditional quality control systems. The main factors that contribute to the quality of the crowdsourced output are the description of the task statements, incentives provided for successful task completion, the processes implemented to detect and mitigate malicious behaviors or low-quality information, and the abilities of the agents who volunteered as workers.

?

A comprehensive survey on quality control in crowdsourcing was conducted by (Florian et all., 2018). This survey concluded that quality control in crowdsourcing is understood to be crucial by all involved and that significant energy has been invested into it, however, there are some key areas that require further improvements to effectively harness the problem-solving capabilities of human intelligence. One of these areas in need of improvement is domain specific services. Currently most crowdsourcing platforms place themselves as providers of technology that manage the crowd and the tasks from a conceptual approach. This requires those who are seeking to use crowdsourcing to possess sufficient knowledge of crowds for optimal requirement structuring for use in crowds. It is recommended that future research be pursued as to how domain-specific providers can assist requestors less familiar with crowds in all phases of the crowdsourcing process.

?

The second area that was emphasized was crowd work regulation and ethics. Due to crowdsourcing being a global event, common rules, practices, and regulations that allow for ethical treatment of the crowd workers is not consistent. Rules and regulations involving taxations, pension, superannuation, and dispute resolutions are still lacking, or inconsistent at best, among the different crowdsourcing providers. As the domain of crowdsourcing continues to grow and mature, it may become more apparent as to how to effectively deal with these issues.

?

Crowdsourcing Case Study of Complex Vehicle Design by Local Motors Inc.

From a system engineering standpoint, decomposition of requirements and task assignment commonly works as a hierarchical flow from the largest complex level to more simple levels of task complexity. The example that was given as a case study to demonstrate this in “Crowdsourcing: A Primer and its Implications for Systems Engineering” (Gerth, and Burnap, 2012d) is a company named Local Motors Inc. Local Motors Inc. worked to develop a vehicle using the crowdsourcing method. The basic system level design of this process is depicted in Exhibit 6. This shows a series of crowd evaluations with internal design processes conducted by experts that were either staffed or contracted.

Exhibit 6. Crowdsource Model for Vehicle Design (Gerth, and Burnap, 2012).

Vehicle requirements is the starting point and, as mentioned before, these can be approximately or closely defined depending on the level of creative liberty the client wishes to grant. These requirements then form the basis for the first crowdsourced task statement. For Local Motors Inc., the first crowdsourced task statement was to design the basic vehicle styling and internal component packaging such as the user interface, seating, and gauge orientation. Once these task statements received task submissions and winning designs were chosen.

?

Many of the requirements for the subsystems and interfaces between the subsystems were designed by internal company experts, however, when Local Motors Inc. wanted to crowdsource a specific subsystem, requirements such as interfaces, packaging constraints, etc. were defined and stated as a challenge or task statement for the crowd to design and or evaluate. The seats were crowdsourced in the following manner. The crowd designed and evaluated the shape, styling, materials, and structure. The company then took the designs and determined the manufacturability of them and then sent the designs to be produced.

?

Local Motors Inc. was able to iteratively implement this method of task statement definition and submission throughout the design process and found that there were many advantages to utilizing crowd evaluation in their design. One of these advantages was the designs that were submitted earlier in the open period were critiqued by the crowd and could then be re-designed based on crowd inputs and resubmitted as improved revisions. Another advantage was that the crowd was able to evaluate far more designs at a faster rate and more cost effectively than an internal review board would have been able to do. Since the crowd had a part in the development of the vehicle, many of them developed a sense of ownership in the final design, which can lead to a faster adoption of the product.

?

Simulation of Crowdsourcing Model Case Study ?

Ultimately, an understanding must be formed around the relationships among a task, the amount of expertise of individuals participating, the number of individuals participating, and the quality of task submissions. Gerth and Burnap (2012e) attempt to answer this through a multi agent-based simulation for the data generation coupled with machine learning methods for the data analysis. Multi agent-based simulation was chosen for its ability to cope with very different models of individuals, ranging from simple entities to complex ones.

It was assumed that one does not have to model all human cognition to make the agent simulation sufficiently accurate. The method that was used involved the assumption of certain psychological trends to account for some of the variability, and the rest of the variability would be machine learned. An example was given that supposes that an agent must choose between designs labeled A through Z. The behavioral model based on experience and demographics of an agent might predict the choice of Design T and the data shows that Design T was chosen 40% of the time. This means that the model is not able to account for 60% of the variability. The machine learning techniques can deduce models that explain this 60% discrepancy given enough data (Gerth, and Burnap, 2012f).

?

Exhibit 7. Plot of Simulation Results (Gerth, and Burnap, 2012).

Exhibit 7 shows the results of the simulation with the ratio of crowd error over average individual error being the x axis and the y axis representing the increase in crowd size and decrease in individual expertise in the domain of the task statement. Within the assumptions of this simulation, larger groups of non-experts perform similarly to small groups of experts. This simulation was not totally conclusive, as it made many assumptions and only examined evaluation tasks, however, it does provide enough data to suggest that given simple tasks, it is possible for a large crowd to outperform a dedicated team of experts. It stands to reason that if a task can be decomposed and defined sufficiently, it can be crowdsourced effectively. This can include the tasks of determining system requirements and quality control systems.

?

Conclusion

Crowdsourcing is a method of outsourcing work over a distributed network. This is a concept that has been around for many years, however, with the developments of the internet has experienced new potential use case expansions. Crowdsourcing can be broken down into six essential elements and has various methods with which it may be deployed. This paper analyzed crowdsourcing and its applications in requirements engineering and quality control systems with the intention of understanding its future within these domains.

?

The data collected suggested that it is possible for a large crowd of non-experts to outperform a small crowd of experts. One of the most significant factors in predicting the successful application of crowdsourcing was not necessarily the complexity of the overall system, but rather how the requirements within each task and subtask were decomposed. If these requirements can be sufficiently decomposed and well-defined, they can likely be crowdsourced. With advantages such as the ability to evaluate far more designs at a faster rate and more cost effectively, it can be assumed that crowdsourcing will be the preferred method of system design in any domain in which the requirements can be met.

?

Author profile

Jordan J. Clayton is an aviation project engineer at an electrical contracting company called Hypower inc. He graduated with honors from Florida Atlantic University with a bachelor’s degree in civil engineering and is currently attending the University of Colorado at Colorado Springs to complete his Master of Engineering in space operations. His expected graduation date is Fall of 2023.

?

References

Adepetu, Adedamola, Khaja Altaf Ahmed, Yousif Al Adb, Aaesha Al Zaabi and Davor Svetinovic,

“CrowdREquire: A Requirements Engineering Crowdsourcing Platform” Proceedings from AAAI Technical Report SS-12-06 Wisdom of the Crowd, Masdar Institute of Science and Technology, Abu Dhabi, United Arab Emirates (2012a), pp. 1

Adepetu, Adedamola, Khaja Altaf Ahmed, Yousif Al Adb, Aaesha Al Zaabi and Davor Svetinovic,

“CrowdREquire: A Requirements Engineering Crowdsourcing Platform” Proceedings from AAAI Technical Report SS-12-06 Wisdom of the Crowd, Masdar Institute of Science and Technology, Abu Dhabi, United Arab Emirates (2012b), pp. 2

Blanchard, Benjamin, S., and Wolter J. Fabrycky, “Systems Engineering and Analysis Fifth Edition” Prentice Hall (2011a) pp.38

Blanchard, Benjamin, S., and Wolter J. Fabrycky, “Systems Engineering and Analysis Fifth Edition” Prentice Hall (2011b) pp.40

“DARPA Subterranean Challenge Final Event.” DARPA RSS, www.darpa.mil/news-events/darpa-subterranean-challenge-final-event.