Interactive Tool for Specifying Factor Relationships in Design Structure
A computing device obtains first and second factors for a nesting design and first and second sets of candidate options for the first and second factors, respectively. The computing device also receives a nesting indication that the second factor is nested within the first factor. The computing device generates the nesting design by portioning the first and second sets of candidate options into respective first and second different sets of groups. Based on the nesting indication, the computing device generates an assignment of groups for a design, and also selects options in the nesting design for the first and second factors based on the assignment of groups. The computing device also generates terms for a computer model for a computer simulation of the nesting design, as well as one or more computer-generated assessments of the nesting design based on the computer simulation.
This application claims the benefit of, and priority based on, 35 U.S.C. § 119 to U.S. Provisional Application No. 63/315,039, filed Feb. 28, 2022, and U.S. Provisional Application No. 63/327,447, filed Apr. 5, 2022, the disclosures of each of which are incorporated herein by reference in their entirety.
BACKGROUNDWhen designing and conducting experiments, factors in the experiment can be crossed with one another. For instance, with crossed categorical factors, in a full factorial design every level of one factor is paired with every level of another factor, so that all possible combinations appear in the design. As an example, a design for a measurement systems analysis may have 3 operators tasked with measuring 10 parts with 2 instruments. In a fully crossed design, each operator would measure every part with every instrument. However, in practice, circumstances may arise that prevent a crossed design from being implemented. For example, if the measurement system analysis were conducted at two different sites, it may be impractical to move the operators, parts, and instruments between the different sites. In such cases, nested designs may be used. For instance, operator, part, and instrument factors can be nested within a site factor so that not every site is paired with every other operator, part, and instrument.
SUMMARYIn an example embodiment, the present disclosure provides a computer-program product tangibly embodied in a non-transitory machine-readable storage medium. The computer-program product includes system instructions operable to cause a computing system to obtain a first factor for a nesting design and first set of candidate options for the first factor. The computer-program product also includes instructions operable to cause the computing system to obtain a second factor for the nesting design and a second set of candidate options for the second factor. Additionally, the computer-program product includes instructions operable to cause the computing system to receive, using a graphical user interface, a nesting indication that the second factor is nested within the first factor and generate the nesting design. To generate the nesting design, the computer-program product includes instructions operable to cause the computing system to portion the first set of candidate options into at least two groups of a first set of groups and portion the second set of candidate options into at least two groups of a second set of groups. The first set of groups are different from the second set of groups. Additionally, to generate the nesting design, the computer-program product includes instructions operable to cause the computing system to generate, based on the nesting indication, an assignment of groups for a design. The assignment of groups assigns a first group from the first set of groups for use with a second group of the second set of groups in the nesting design. Additionally, to generate the nesting design, the computer-program product includes instructions operable to cause the computing system to select, based on the assignment of groups, options in the nesting design for the first factor from the first set of candidate options, and select, based on the assignment of groups, options in the nesting design for the second factor from the second set of candidate options. The computer-program product also includes instructions operable to further cause the computing system to generate terms for a computer model for a computer simulation of the nesting design. An amount of terms in the computer model are constrained by the computing system according to the nesting indication that the second factor is nested within the first factor. The computer-program product also includes instructions operable to cause the computing system to generate one or more computer-generated assessments of the nesting design based on the computer simulation of the nesting design according to the computer model.
Embodiments disclosed herein also include a corresponding apparatus and method. For example, in another example embodiment, the present disclosure provides a computer-implemented method. The method comprises obtaining a first factor for a nesting design and first set of candidate options for the first factor, obtaining a second factor for the nesting design and a second set of candidate options for the second factor, and receiving, using a graphical user interface, a nesting indication that the second factor is nested within the first factor. The method also comprises a computing system generating the nesting design by portioning the first set of candidate options into at least two groups of a first set of groups and portioning the second set of candidate options into at least two groups of a second set of groups. The first set of groups are different from the second set of groups. The method also comprises the computing system generating the nesting design by generating, based on the nesting indication, an assignment of groups by the computing system for a design. The assignment of groups assigns a first group from the first set of groups for use with a second group of the second set of groups in the nesting design. The method also comprises the computing system generating the nesting design by selecting, based on the assignment of groups, options in the nesting design for the first factor from the first set of candidate options and selecting, based on the assignment of groups, options in the nesting design for the second factor from the second set of candidate options. The method also comprises the computing system generating terms for a computer model for a computer simulation of the nesting design. In the example embodiment, an amount of terms in the computer model are constrained by the computing system according to the nesting indication that the second factor is nested within the first factor. The method also comprises the computing system generating one or more computer-generated assessments of the nesting design based on the computer simulation of the nesting design according to the computer model.
In another example embodiment, the present disclosure provides a computing device comprising a processor and a memory containing instructions executable by the processor. In this example embodiment, executing the instructions by the processor configures the computing device to obtain a first factor for a nesting design and first set of candidate options for the first factor, obtain a second factor for the nesting design and a second set of candidate options for the second factor, and receive, using a graphical user interface, a nesting indication that the second factor is nested within the first factor. Additionally, executing the instructions by the processor configures the computing device to generate the nesting design by portioning the first set of candidate options into at least two groups of a first set of groups and portioning the second set of candidate options into at least two groups of a second set of groups. The first set of groups and the second set of groups are different. Executing the instructions by the processor configures the computing device to generate the nesting design by generating, based on the nesting indication, an assignment of groups for a design. The assignment of groups assigns a first group from the first set of groups for use with a second group of the second set of groups in the nesting design. Executing the instructions by the processor configures the computing device to generate the nesting design by selecting, based on the assignment of groups, options in the nesting design for the first factor from the first set of candidate options and selecting, based on the assignment of groups, options in the nesting design for the second factor from the second set of candidate options. Executing the instructions by the processor configures the computing device to generate terms for a computer model for a computer simulation of the nesting design. An amount of terms in the computer model are constrained by the computing device according to the nesting indication that the second factor is nested within the first factor. Executing the instructions by the processor configures the computing device to generate one or more computer-generated assessments of the nesting design based on the computer simulation of the nesting design according to the computer model.
Other features and aspects of example embodiments are presented below in the Detailed Description when read in connection with the drawings presented with this application.
In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of embodiments of the technology. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be restrictive.
The ensuing description provides example embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the example embodiments will provide those skilled in the art with an enabling description for implementing an example embodiment. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the technology as set forth in the appended claims.
Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.
Also, it is noted that individual embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed but could have additional operations not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to a return of the function to the calling function or the main function.
Systems depicted in some of the figures may be provided in various configurations. In some embodiments, the systems may be configured as a distributed system where one or more components of the system are distributed across one or more networks in a cloud computing system.
Network devices and client devices can transmit a communication over a network 100. Network 100 may include one or more of different types of networks, including a wireless network, a wired network, or a combination of a wired and wireless network. Examples of suitable networks include the Internet, a personal area network, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), cloud network, or a cellular network. A wireless network may include a wireless interface or combination of wireless interfaces. As an example, a network in the one or more networks may include a short-range communication channel, such as a Bluetooth or a Bluetooth Low Energy channel. A wired network may include a wired interface. The wired and/or wireless networks may be implemented using routers, access points, base stations, bridges, gateways, or the like, to connect devices in the network. The one or more networks can be incorporated entirely within or can include an intranet, an extranet, or a combination thereof. In one embodiment, communications between two or more systems and/or devices can be achieved by a secure communications protocol, such as secure sockets layer (SSL) or transport layer security (TLS), or other available protocols such as according to an Open Systems Interaction model. In addition, data and/or transactional details may be encrypted. Networks may include other devices for infrastructure for the network. For example, a cloud network may include cloud infrastructure system on demand. As another example, one or more client devices may utilize an Internet of Things (IoT) infrastructure where things (e.g., machines, devices, phones, sensors) can be connected to networks and the data from these things can be collected and processed within the things and/or external to the things. IoT may be implemented with various infrastructure such as for accessibility (technologies that get data and move it), embed-ability (devices with embedded sensors), and IoT services. Industries in the IoT space may include automotive (connected car), manufacturing (connected factory), smart cities, energy and retail.
Network devices and client devices can be different types of devices or components of devices. For example, client device 140 is shown as a laptop and balancer 160 is shown as a processor. Client devices and network devices could be other types of devices or components of other types of devices such as a mobile phone, laptop computer, tablet computer, temperature sensor, motion sensor, and audio sensor. Additionally, or alternatively, the network devices may be or include sensors that are sensitive to detecting aspects of their environment. For example, the network devices may include sensors such as water sensors, power sensors, electrical current sensors, chemical sensors, optical sensors, pressure sensors, geographic or position sensors (e.g., GPS), velocity sensors, acceleration sensors, and flow rate sensors. Examples of characteristics that may be sensed include force, torque, load, strain, position, temperature, air pressure, fluid flow, chemical properties, resistance, electromagnetic fields, radiation, irradiance, proximity, acoustics, moisture, distance, speed, vibrations, acceleration, electrical potential, and electrical current. The sensors may be mounted to various components used as part of a variety of different types of systems (e.g., an oil drilling operation). The network devices may detect and record data related to the environment that it monitors, and transmit that data to network 100.
As noted, one type of system that may include various sensors that collect data to be processed and/or transmitted to a computing environment (not shown) according to certain embodiments includes an oil drilling system. For example, the one or more drilling operation sensors may include, for example, surface sensors that measure a standpipe pressure, a surface torque, and a rotation speed of a drill pipe, and downhole sensors that measure a rotation speed of a bit and fluid densities. Besides the raw data collected directly by the sensors, other data may include parameters either developed by the sensors or assigned to the system by a client or other controlling device. For example, one or more drilling operation control parameters may control settings such as a mud motor speed to flow ratio, a bit diameter, a predicted formation top, seismic data, weather data, etc. Other data may be generated using physical models such as an earth model, a weather model, a seismic model, a bottom hole assembly model, a well plan model, an annular friction model, etc. In addition to sensor and control settings, predicted outputs, of for example, the rate of penetration and pump pressure may also be stored and used for modeling, prediction, or classification.
In another example, another type of system that may include various sensors that collect data to be processed and/or transmitted to a computing environment according to certain embodiments includes a home automation or similar automated network in a different environment, such as an office space, school, public space, sports venue, or a variety of other locations. Network devices in such an automated network may include network devices that allow a user to access, control, and/or configure various home appliances located within the user's home (e.g., a television, radio, light, fan, humidifier, sensor, microwave, iron, and/or the like), or outside of the user's home (e.g., exterior motion sensors, exterior lighting, garage door openers, sprinkler systems, or the like). For example, network device or client device may include a home automation switch that may be coupled with a home appliance. In another embodiment, a network or client device can allow a user to access, control, and/or configure devices, such as office-related devices (e.g., copy machine, printer, or fax machine), audio and/or video related devices (e.g., a receiver, a speaker, a projector, a DVD player, or a television), media-playback devices (e.g., a compact disc player, a CD player, or the like), computing devices (e.g., a home computer, a laptop computer, a tablet, a personal digital assistant (PDA), a computing device, or a wearable device), lighting devices (e.g., a lamp or recessed lighting), devices associated with a security system, devices associated with an alarm system, devices that can be operated in an automobile (e.g., radio devices, navigation devices), and/or the like. Data may be collected from such various sensors in raw form, or data may be processed by the sensors to create parameters or other data either developed by the sensors based on the raw data or assigned to the system by a client or other controlling device.
In another example, another type of system that may include various sensors that collect data to be processed and/or transmitted to a computing environment (e.g., computing environment or another computing environment not shown) according to certain embodiments includes a manufacturing environment (e.g., manufacturing products or energy). A variety of different network devices may be included in an energy pool, such as various devices within one or more power plants, energy farms (e.g., wind farm, and solar farm) energy storage facilities, factories, homes and businesses of consumers. One or more of such devices may include one or more sensors that detect energy gain or loss, electrical input or output or loss, and a variety of other efficiencies. These sensors may collect data to inform users of how the energy pool, and individual devices within the pool, may be functioning and how they may be made more efficient. In a manufacturing environment, image data can be taken of the manufacturing process or other readings of manufacturing equipment. For example, in a semiconductor manufacturing environment, images can be used to track, for example, process points (e.g., movement from a bonding site to a packaging site), and process parameters (e.g., bonding force, electrical properties across a bond of an integrated circuit).
Network device sensors may also perform processing on data it collects before transmitting the data to a computing environment, or before deciding whether to transmit data to a computing environment. For example, network devices may determine whether data collected meets certain rules, for example by comparing data or values calculated from the data and comparing that data to one or more thresholds. The network device may use this data and/or comparisons to determine if the data should be transmitted to a computing environment for further use or processing.
Devices in computing environment 114 may include specialized computers, servers, or other machines that are configured to individually and/or collectively process large amounts of data (e.g., using a session pool 102). The computing environment 114 may also include storage devices (e.g., data stores 120) that include one or more databases of structured data, such as data organized in one or more hierarchies, or unstructured data. The databases may communicate with the processing devices within computing environment 114 to distribute data to them and store data used in the computing environment 114. Computing environment 114 may collect, analyze and/or store data from or pertaining to communications, client device operations, client rules, and/or user-associated actions stored at one or more devices in computing environment 114. Such data may influence communication routing to the devices within computing environment 114, and how data is stored or processed within computing environment 114, among other actions.
Network 100 may also include one or more network-attached data stores 120. Network-attached data stores 120 are used to store data to be processed by the computing environment 114 as well as any intermediate or final data generated by the computing system in non-volatile memory. For instance, data stores 120 can perform functions such as writing and copying data and can provide data storage for network functions such as sessions, authorization, publishing and retrieving packages. In certain embodiments, the configuration of the computing environment 114 allows its operations to be performed such that intermediate and final data results can be stored solely in volatile memory (e.g., RAM), without a requirement that intermediate or final data results be stored to non-volatile types of memory (e.g., disk). This can be useful in certain situations, such as when the computing environment 114 receives ad hoc queries from a user and when responses, which are generated by processing large amounts of data, need to be generated on-the-fly. In this non-limiting situation, the computing environment 114 may be configured to retain the processed information within memory so that responses can be generated for the user at different levels of detail as well as allow a user to interactively query against this information.
Network-attached data stores 120 may store a variety of different types of data organized in a variety of different ways and from a variety of different sources. For example, network-attached data stores 120 may include storage other than primary storage located within computing environment 114 that is directly accessible by processors located therein. Network-attached data stores 120 may include secondary, tertiary, auxiliary, or back-up storage (e.g., data storage 120B), such as large hard drives, servers, and virtual memory, among other types. Storage devices may include portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing and containing data (e.g., computer a machine-readable storage medium or computer-readable storage medium such as computer readable medium 210 in
Furthermore, the data stores may hold a variety of different types of data. For example, network-attached data stores 120 may hold unstructured (e.g., raw) data, such as manufacturing data (e.g., a database containing records identifying products being manufactured with parameter data for each product, such as performance metrics or criteria) or product sales databases (e.g., a database containing individual data records identifying details of individual product performance).
The unstructured data may be presented to the computing environment 114 in different forms such as a flat file or a conglomerate of data records and may have data values and accompanying time stamps. The computing environment 114 may be used to analyze the unstructured data in a variety of ways to determine the best way to structure (e.g., hierarchically) that data, such that the structured data is tailored to a type of further analysis on the data. For example, after being processed, the unstructured time stamped data may be aggregated by time (e.g., into daily time period units) to generate time series data and/or structured hierarchically according to one or more dimensions (e.g., parameters, attributes, and/or variables). For example, data may be stored in a hierarchical data structure, such as a ROLAP OR MOLAP database, or may be stored in another tabular form, such as in a flat-hierarchy form.
Other devices can further be used to influence communication routing and/or processing between devices within computing environment 114 and with devices outside of computing environment 114. For example, as shown in
In addition to computing environment 114 collecting data (e.g., as received from network devices, such as sensors, and client devices or other sources) to be processed as part of a big data analytics project, it may also receive data in real time as part of a streaming analytics environment. As noted, data may be collected using a variety of sources as communicated via different kinds of networks or locally. Such data may be received on a real-time streaming basis. For example, network devices may receive data periodically from sensors as the sensors continuously sense, monitor and track changes in their environments. Devices within computing environment 114 may also perform pre-analysis on data it receives to determine if the data received should be processed as part of an ongoing project. The data received and collected by computing environment 114, no matter what the source or method or timing of receipt, may be processed over a period of time for a client to determine results data based on the client's needs and rules.
Session pool 102 includes one or more worker nodes (e.g., worker node 110A). Shown in
The pool manager 104 may connect with other devices of network 100 or an external device (e.g., a pool user, such as a server or computer). For example, a server or computer may connect to pool manager 104 and may transmit a project or job to the node. The project may include a data set. The data set may be of any size. Once the pool manager 104 receives such a project including a large data set, the pool manager 104 may distribute the data set or projects related to the data set to be performed by worker nodes 110. Alternatively, for a project including a large data set, the data set may be received or stored by a machine other than a pool manager 104 or worker node 110 (e.g., a Hadoop data node).
Pool manager may maintain knowledge of the status of the worker nodes 110 in the session pool 102 (i.e., status information), accept work requests from clients, subdivide the work across worker nodes 110, and coordinate the worker nodes 110, among other responsibilities. Worker nodes 110 may accept work requests from a pool manager 104 and provide the pool manager 104 with results of the work performed by the worker nodes 110. A session pool 102 may be started from a single node (e.g., a machine, computer, server, etc.). This first node may be assigned or may start as the primary pool manager 104 that will control any additional nodes that enter the session pool 102.
When a project is submitted for execution (e.g., by a client or a pool manger 104), it may be assigned to a set of nodes. After the nodes are assigned to a project, a data structure (i.e., a communicator) may be created. The communicator may be used by the project for information to be shared between the project code running on each node. A communication handle may be created on each node. A handle, for example, is a reference to the communicator that is valid within a single process on a single node, and the handle may be used when requesting communications between nodes.
A pool manager may be designated as the primary pool manager among multiple pool managers. A server, computer or other external device may connect to the primary pool manager. Once the pool manager receives a project, the primary pool manager may distribute portions of the project to its worker nodes for execution. For example, when a project is initiated on session pool 102, primary pool manager 104 controls the work to be performed for the project to complete the project as requested or instructed. The primary pool manager may distribute work to the worker nodes 110 based on various factors, such as which subsets or portions of projects may be completed most efficiently and in the correct amount of time. For example, a worker node may perform analysis on a portion of data that is already local (e.g., stored on) the worker node. The primary pool manager also coordinates and processes the results of the work performed by each worker node after each worker node executes and completes its job. For example, the primary pool manager may receive a result from one or more worker nodes, and the pool manager may organize (e.g., collect and assemble) the results received and compile them to produce a complete result for the project received from the end user.
Any remaining pool manager (not shown) may be assigned as backup pool manager for the project. In an embodiment, backup pool manager may not control any portion of the project. Instead, backup pool manager may serve as a backup for the primary pool manager and take over as primary pool manager if the primary pool manager were to fail.
To add another node or machine to the session pool 102, the primary pool manager may open a pair of listening sockets, for example. A socket may be used to accept work requests from clients, and the second socket may be used to accept connections from other pool nodes. The primary pool manager may be provided with a list of other nodes (e.g., other machines, computers, servers) that will participate in the pool, and the role that each node will fill in the pool. Upon startup of the primary pool manager (e.g., the first node on the pool), the primary pool manager may use a network protocol to start the server process on every other node in the session pool 102. Command line parameters, for example, may inform each node of one or more pieces of information, such as: the role that the node will have in the pool, the host name of the primary pool manager, and the port number on which the primary pool manager is accepting connections from peer nodes. The information may also be provided in a configuration file, transmitted over a secure shell tunnel, and recovered from a configuration server. While the other machines in the pool may not initially know about the configuration of the pool, that information may also be sent to each other node by the primary pool manager. Updates of the pool information may also be subsequently sent to those nodes.
For any pool manager other than the primary pool manager added to the pool, the pool manager may open multiple sockets. For example, the first socket may accept work requests from clients, the second socket may accept connections from other pool members, and the third socket may connect (e.g., permanently) to the primary pool manager. When a pool manager (e.g., primary pool manager) receives a connection from another pool manager, it first checks to see if the peer node is in the list of configured nodes in the pool. If it is not on the list, the pool manager may clear the connection. If it is on the list, it may then attempt to authenticate the connection. If authentication is successful, the authenticating node may transmit information to its peer, such as the port number on which a node is listening for connections, the host name of the node, and information about how to authenticate the node, among other information. When a node, such as the new pool manager, receives information about another active node, it will check to see if it already has a connection to that other node. If it does not have a connection to that node, it may then establish a connection to that pool manager.
Any worker node added to the pool may establish a connection to the primary pool manager and any other pool manager on the pool. After establishing the connection, it may authenticate itself to the pool (e.g., any pool manager, including both primary and backup, or a server or user controlling the pool). After successful authentication, the worker node may accept configuration information from the pool manager.
When a node joins a session pool 102 (e.g., when the node is powered on or connected to an existing node on the pool or both), the node is assigned (e.g., by an operating system of the pool) an identifier (e.g., a universally unique identifier (UUID)). This identifier may help other nodes and external entities (devices, users, etc.) to identify the node and distinguish it from other nodes. When a node is connected to the pool, the node may share its identifier with the other nodes in the pool. Since each node may share its identifier, each node may know the identifier of every other node on the pool. Identifiers may also designate a hierarchy of each of the nodes (e.g., backup pool manager) within the pool. For example, the identifiers of each of the backup pool manager may be stored in a list of backup pool manager to indicate an order in which the backup pool manager will take over for a failed primary pool manager to become a new primary pool manager. However, a hierarchy of nodes may also be determined using methods other than using the unique identifiers of the nodes. For example, the hierarchy may be predetermined, or may be assigned based on other predetermined factors.
The pool may add new machines at any time (e.g., initiated from any pool manager). Upon adding a new node to the pool, the pool manager may first add the new node to its table of pool nodes. The pool manager may also then notify every other pool manager about the new node. The nodes receiving the notification may acknowledge that they have updated their configuration information.
Primary pool manager 104 may, for example, transmit one or more communications to backup pool manager or other control or worker nodes within the session pool 102). Such communications may be sent using protocols such as periodically, at fixed time intervals, or between known fixed stages of the project's execution. The communications transmitted by primary pool manager 104 may be of varied types and may include a variety of types of information. For example, primary pool manager 104 may transmit snapshots (e.g., status information) of the session pool 102 so that backup pool manager 104 always has a recent snapshot of the session pool 102. The snapshot or pool status may include, for example, the structure of the pool (including, for example, the worker nodes in the pool, unique identifiers of the nodes, or their relationships with the primary pool manager) and the status of a project (including, for example, the status of each worker node's portion of the project). The snapshot may also include analysis or results received from worker nodes in the session pool 102. The backup pool manager may receive and store the backup data received from the primary pool manager. The backup pool manager may transmit a request for such a snapshot (or other information) from the primary pool manager, or the primary pool manager may send such information periodically to the backup pool manager.
As noted, the backup data may allow the backup pool manager to take over as primary pool manager if the primary pool manager fails without requiring the pool to start the project over from scratch. If the primary pool manager fails, the backup pool manager that will take over as primary pool manager may retrieve the most recent version of the snapshot received from the primary pool manager and use the snapshot to continue the project from the stage of the project indicated by the backup data. This may prevent failure of the project as a whole.
A backup pool manager may use various methods to determine that the primary pool manager has failed. In one example of such a method, the primary pool manager may transmit (e.g., periodically) a communication to the backup pool manager that indicates that the primary pool manager is working and has not failed, such as a heartbeat communication. The backup pool manager may determine that the primary pool manager has failed if the backup pool manager has not received a heartbeat communication for a certain predetermined period of time. Alternatively, a backup pool manager may also receive a communication from the primary pool manager itself (before it failed) or from a worker node that the primary pool manager has failed, for example because the primary pool manager has failed to communicate with the worker node.
Different methods may be performed to determine which backup pool manager of a set of backup pool manager will take over for failed primary pool manager 104 and become the new primary pool manager. For example, the new primary pool manager may be chosen based on a ranking or “hierarchy” of backup pool manager based on their unique identifiers. In an alternative embodiment, a backup pool manager may be assigned to be the new primary pool manager by another device in the session pool 102 or from an external device (e.g., a system infrastructure or an end user, such as a server or computer, controlling the session pool 102). In another alternative embodiment, the backup pool manager that takes over as the new primary pool manager may be designated based on bandwidth or other statistics about the session pool 102.
A worker node within the session pool 102 may also fail. If a worker node fails, work being performed by the failed worker node may be redistributed amongst the operational worker nodes. In an alternative embodiment, the primary pool manager may transmit a communication to each of the operable worker nodes still on the session pool 102 that each of the worker nodes should purposefully fail also. After each of the worker nodes fail, they may each retrieve their most recent saved checkpoint of their status and re-start the project from that checkpoint to minimize lost progress on the project being executed.
While each device in
Processor 208 executes instructions (e.g., stored at the computer-readable medium 210). The instructions can be carried out by a special purpose computer, logic circuits, or hardware circuits. In one or more embodiments, processor 208 is implemented in hardware and/or firmware. Processor 208 executes an instruction, meaning it performs or controls the operations called for by that instruction. The term “execution” is the process of running an application or the carrying out of the operation called for by an instruction. The instructions can be written using one or more programming language, scripting language, assembly language, etc. Processor 208 in one or more embodiments can retrieve a set of instructions from a permanent memory device and copy the instructions in an executable form to a temporary memory device that is generally some form of RAM, for example. Processor 208 operably couples with components of computing device 202 (e.g., input/output interface 204 and with computer readable medium 210) to receive, to send, and to process information.
For instance, in one or more embodiments, computing device 202 sends and/or receives information from one or more of databases 230, cloud sources 232, application programming interfaces 236 (API's), graphical user interfaces 240 (GUIs), printers 242, webpages 244, and computing systems 246. The input/output interface 204 may be configured to receive languages 238 (e.g., to communicate with other computing systems 246) or specific electronic files or documents 234 (e.g., inputs for building models or designing experiments). The input/output interface 204 may be a single interface (e.g., an output interface only to output reports to a printer 242), multiple interface (e.g., a graphical user interface 240 may be interactive and send and receive data over input/output interface 204), or a set of interfaces (e.g., to connect with multiple devices).
In one or more embodiments, computer-readable medium 210 stores instructions for execution by processor 208. In one or more embodiments, one or more applications stored on computer-readable medium 210 are implemented in software (e.g., computer-readable and/or computer-executable instructions) stored in computer-readable medium 210 and accessible by processor 208 for execution of the instructions.
System 300 includes data processing nodes (e.g., control node 302 and worker node 310). Control node 302 and worker node 310 can include multi-core data processors. Each control node 302 and worker node 310 in this example includes a grid-enabled software component (GESC) 320 that executes on the data processor associated with that node and interfaces with buffer memory 322 also associated with that node. Each control node 302 and worker node 310 in this example includes a database management software (DBMS) 328 that executes on a database server (not shown) at control node 302 and on a database server (not shown) at worker node 310.
Each control node 302 and worker node 310 in this example also includes a data storage 324. Data storage 324, similar to network-attached data stores 120 in
Each control node 302 and worker node 310 in this example also includes a user-defined function (UDF) 326. The UDF 326 provides a mechanism for the DBMS 328 to transfer data to or receive data from the database stored in the data storage 324 that are managed by the DBMS. For example, UDF 326 can be invoked by the DBMS 328 to provide data to the GESC 320 for processing. The UDF 326 may establish a socket connection (not shown) with the GESC 320 to transfer the data. Alternatively, the UDF 326 can transfer data to the GESC 320 by writing data to shared memory accessible by both the UDF 326 and the GESC 320.
The GESC 320 at the control node 302 and worker node 310 may be connected via a network. Therefore, control node 302 and worker node 310 can communicate with each other via the network using a predetermined communication protocol such as, for example, the Message Passing Interface (MPI). Each GESC 320 can engage in point-to-point communication with the GESC at another node or in collective communication with multiple GESCs via the network. The GESC 320 at each node may contain identical (or nearly identical) software instructions. Each control node 302 and worker node 310 may be configured to operate as either a pool manager or a worker node. The GESC 320B at the control node 302 can communicate, over a communication path 352, with a client device 330. More specifically, control node 302 may communicate with client application 332 hosted by the client device 330 to receive queries and to respond to those queries after processing large amounts of data.
DBMS 328 may control the creation, maintenance, and use of database or data structure (not shown) within control node 302 and worker node 310. The database may organize data stored in data storage 324. The DBMS 328 at control node 302 may accept requests for data and transfer the appropriate data for the request. With such a process, collections of data may be distributed across multiple physical locations. In this example, each control node 302 and worker node 310 stores a portion of the total data managed by the management system in its associated data storage 324.
Furthermore, the DBMS 328 may be responsible for protecting against data loss using replication techniques. Replication includes providing a backup copy of data stored on one node on one or more other nodes. Therefore, if one node fails, the data from the failed node can be recovered from a replicated copy residing at another node. Data or status information for each node in the session pool 102 may also be shared with each node on the pool.
For example, data access operations 402 can be used for accessing data from different sources (e.g., importing and/or reading Excel files, flat files, relational databases, APIs, R, Python, and SAS® files and databases). For instance, data can be imported for data visualization, exploration and analysis. Data can be formatted or optimized. For instance, data blending and cleanup operations 404 can be used to remove complexity (e.g., in text, images and functions data) and for screening data (e.g., screening data for outliers, entry errors, missing values and other inconsistencies that can compromise data analysis). This can be useful for visual and interactive tools. Data can also be transformed, blended, grouped, filtered, merged into a single table or into subsets, or otherwise arranged for a particular scenario.
In one or more embodiments, one or more applications 400 include data exploration and visualization operations 406 that can be used to support plot and profiler tools. For instance, plot tools can be used to create data plots (e.g., to plot data to spot patterns and patterns that do not fit a trend). Some example plots include bubble plots, scatter plots (matrix and 3D), parallel plots, cell plots, contour plots, ternary plots, and surface plots. Profilers are tools that can be used to create a specialized set of plots in which changing one plot changes the other plots. For instance, profiling is an approach to generate visualizations of response surfaces by seeing what would happen if a user changed just one or two factors at a time. Profiler tools can be used to create interactive profiles of data (e.g., to explore and graph data dynamically and uncover hidden relationships between graphed data or interface with linked data, to interpret and understand the fit of equations to data, and to find factor values to optimize responses). Some example profiler tools include prediction profiler, contour profiler, surface profiler, mixture profiler, custom profiler, and excel profiler. A prediction profiler can be used to show vertical slices across each factor, holding other factors at a current value. A contour profiler allows horizontal slices showing contour lines for two factors at a time. A surface profiler generates three-dimensional plots for two factors at a time, or contour surface plot for 3 factors at a time. A mixture profiler is a contour profiler for mixture of factors. A custom profiler is a numerical optimizer. An excel profiler allows for visualization of models or formulas stored in electronic worksheets. Accordingly, profiler tools can allow for one or more of simulation, surface visualization, optimization, and desirability studies. Graphs (e.g., from plot or profiler tools) can be exported to electronic or print reports for presenting findings. Further, data exploration and visualization operations 406 can include text exploration such as computer extraction of symbols, characters, words and phrases; or computer visualization such as to organize symbols, characters, words and phrases to uncover information regarding a text or classify the text.
In one or more embodiments, one or more applications 400 include data analysis and modeling operations 408 can be used to analyze one or many variables or factors in linked analysis. Analysis results may be linked with specific graphs designed for different types of data or metrics (e.g., graphs related to histograms, regression modeling and distribution fitting). Data analysis and modeling can be performed real-time (or just-in-time). For instance, applications 400 can included statistical modeling operations 410. For instance, statistical modeling operations 410 can be used for a diversity of modeling tasks such as univariate, multivariate and multifactor. Data can be transformed from its collected form (e.g., text or functional form) and data can be used for building models for better insights (e.g., discovery trends or patterns in data). As another example, one or more applications 400 can include predictive modeling and machine learning operations 412 to build models using predictive modeling techniques, such as regression, neural networks and decision trees. The operations 412 can be used to fit multiple predictive models and determine the best performing model with model screening. Validation (e.g., cross-validation and k-fold cross-validation) can be used (e.g., to prevent over-fitting or to select a best model). Machine learning methods can be used by the user without having to write code and tune algorithms. Examples of machine learning techniques are described in more detail with respect to
In one or more embodiments, one or more applications 400 include design of experiments (DOE) operations 414 used to create designs for experiments that provide test conditions for one or more factors tested in the experiment. For example, the design of experiments operations 414 can be used to create optimally designed experiments, efficient experiments to meet constraints, process limitations and budget, and/or screening designs to untangle important effects between multiple factors. DOE operations 414 can also be used for evaluating designs (e.g., design diagnostic measures such as efficiency metrics).
In one or more embodiments, one or more applications 400 include quality and process engineering operations 416 to track and visualize quality and processes. For instance, the quality and process engineering operations 416 can generate charts to explore root causes of quality or process problems (e.g., causes of variation in manufacturing processes and drill down into problem processes). Additionally, or alternatively, they can be used to generate notifications for metrics that exceed a threshold such as an out-of-control signal or a control chart warning. Additionally, or alternatively, they can be used to study the capability and performance of one or more variables to identify processes that are not meeting user-defined goals. Objective data from processes or consumer data can be used to release better products and react to market trends.
In one or more embodiments, one or more applications 400 include reliability analysis operations 418. For example, in manufacturing, reliability analysis tools can be used to prevent failure, improve warranty or product performance, find and address important design vulnerabilities, and pinpoint defects in materials or processes. Reliability analysis tools can also be used to determine how to reduce or improve these issues (e.g., by identifying trends and outliers in data and model predictions). What-if Analysis operations 422 can be used to demonstrate patterns of predicted responses and the effect of each factor on the response with scenario analysis. For example, a graphical user interface can be used for a user to put in different inputs, assumptions or constraints for a system and observe responses or effects. For instance, in a measurement system analysis analyzing whether parts would be in-specification, different estimated variances between parts and operators testing the parts could be varied to determine the effect on modeled output for the measurement system analysis.
In one or more embodiments, one or more applications 400 include automation and scripting operations 420. For example, automation can allow code-free access for a user to automation routines all the way up to completely customized applications (e.g., code free access to SAS®, MATLAB®, Python® and R routines). For example, a design created for experiments can be automated such that automatic testing is performed for the design.
In one or more embodiments, one or more applications 400 include operations for greater user control and interaction. For instance, customization operations 424 can be used for user customization (e.g., mass customizations, and customizations of graphics, statistics, and default views). As another example, content organization operations 426 can be used to organize data (e.g., translate statistical results to a simplified view to communicate findings and organize, summarize, and document content to better aid the accountability and reproducibility of projects). As another example, the communicating results operations 428 can be used for presentation of results, models, or other output from one or more applications 400 (e.g., presented in print, graphical user interface, or web-based versions).
In one or more embodiments, fewer, different, and additional components can be incorporated into computing device 202. In one or more embodiments, the input/output interface has more than one interface that uses the same or different interface technology.
In one or more embodiments, the one or more applications 400 can be integrated with other analytic or computing tools not specifically shown here. For instance, one or more applications are implemented using or integrated with one or more software tools such as JMP®, Base SAS, SAS® Enterprise Miner™, SAS/STAT®, SAS® High Performance Analytics Server, SAS® Visual Data Mining and Machine Learning, SAS® LASR™ SAS® In-Database Products, SAS® Scalable Performance Data Engine, SAS® Cloud Analytic Services, SAS/OR®, SAS/ETS®, SAS® Inventory Optimization, SAS® Inventory Optimization Workbench, SAS® Visual Analytics, SAS® Viya™, SAS In-Memory Statistics for Hadoop®, SAS® Forecast Server, and SAS/IML®.
One or more embodiments are useful for generating and using machine-learning models.
Different machine-learning models may be used interchangeably to perform a task. Examples of tasks that can be performed at least partially using machine-learning models include various types of scoring; bioinformatics; cheminformatics; software engineering; fraud detection; customer segmentation; generating online recommendations; adaptive websites; determining customer lifetime value; search engines; placing advertisements in real time or near real time; classifying DNA sequences; affective computing; performing natural language processing and understanding; object recognition and computer vision; robotic locomotion; playing games; optimization and metaheuristics; detecting network intrusions; medical diagnosis and monitoring; or predicting when an asset, such as a machine, will need maintenance.
Any number and combination of tools can be used to create machine-learning models. Examples of tools for creating and managing machine-learning models can include SAS® Enterprise Miner, SAS® Rapid Predictive Modeler, and SAS® Model Manager, SAS Cloud Analytic Services (CAS)®, SAS Viya® of all which are by SAS Institute Inc. of Cary, N.C.
Machine-learning models construction can be at least partially automated (e.g., with little or no human involvement) in a training process. During training, input data can be iteratively supplied to a machine-learning model to enable the machine-learning model to identify patterns related to the input data or to identify relationships between the input data and output data. With training, the machine-learning model can be transformed from an untrained state to a trained state. Input data can be split into one or more training sets and one or more validation sets, and the training process may be repeated multiple times. The splitting may follow a k-fold cross-validation rule, a leave-one-out-rule, a leave-p-out rule, or a holdout rule. An overview of training and using a machine-learning model is described below with respect to the flow chart of
In block 504, training data is received. In some examples, the training data is received from a remote database or a local database, constructed from various subsets of data, or input by a user. The training data can be used in its raw form for training a machine-learning model or pre-processed into another form, which can then be used for training the machine-learning model. For example, the raw form of the training data can be smoothed, truncated, aggregated, clustered, or otherwise manipulated into another form, which can then be used for training the machine-learning model.
In block 506, a machine-learning model is trained using the training data. The machine-learning model can be trained in a supervised, unsupervised, or semi-supervised manner. In supervised training, each input in the training data is correlated to a desired output. This desired output may be a scalar, a vector, or a different type of data structure such as text or an image. This may enable the machine-learning model to learn a mapping between the inputs and desired outputs. In unsupervised training, the training data includes inputs, but not desired outputs, so that the machine-learning model has to find structure in the inputs on its own. In semi-supervised training, only some of the inputs in the training data are correlated to desired outputs.
In block 508, the machine-learning model is evaluated. For example, an evaluation dataset can be obtained, for example, via user input or from a database. The evaluation dataset can include inputs correlated to desired outputs. The inputs can be provided to the machine-learning model and the outputs from the machine-learning model can be compared to the desired outputs. If the outputs from the machine-learning model closely correspond with the desired outputs, the machine-learning model may have a high degree of accuracy. For example, if 90% or more of the outputs from the machine-learning model are the same as the desired outputs in the evaluation dataset, the machine-learning model may have a high degree of accuracy. Otherwise, the machine-learning model may have a low degree of accuracy. The 90% number is an example only. A realistic and desirable accuracy percentage is dependent on the problem and the data.
In some examples, if the machine-learning model has an inadequate degree of accuracy for a particular task, the process can return to block 506, where the machine-learning model can be further trained using additional training data or otherwise modified to improve accuracy. If the machine-learning model has an adequate degree of accuracy for the particular task, the process can continue to block 510.
In block 510, new data is received. In some examples, the new data is received from a remote database or a local database, constructed from various subsets of data, or input by a user. The new data may be unknown to the machine-learning model. For example, the machine-learning model may not have previously processed or analyzed the new data.
In block 512, the trained machine-learning model is used to analyze the new data and provide a result. For example, the new data can be provided as input to the trained machine-learning model. The trained machine-learning model can analyze the new data and provide a result that includes a classification of the new data into a particular class, a clustering of the new data into a particular group, a prediction based on the new data, or any combination of these.
In block 514, the result is post-processed. For example, the result can be added to, multiplied with, or otherwise combined with other data as part of a job. As another example, the result can be transformed from a first format, such as a time series format, into another format, such as a count series format. Any number and combination of operations can be performed on the result during post-processing.
A more specific example of a machine-learning model is the neural network 600 shown in
The neurons and connections between the neurons can have numeric weights, which can be tuned during training. For example, training data can be provided to the input layer 602 of the neural network 600, and the neural network 600 can use the training data to tune one or more numeric weights of the neural network 600. In some examples, the neural network 600 can be trained using backpropagation. Backpropagation can include determining a gradient of a particular numeric weight based on a difference between an actual output of the neural network 600 and a desired output of the neural network 600. Based on the gradient, one or more numeric weights of the neural network 600 can be updated to reduce the difference, thereby increasing the accuracy of the neural network 600. This process can be repeated multiple times to train the neural network 600. For example, this process can be repeated hundreds or thousands of times to train the neural network 600.
In some examples, the neural network 600 is a feed-forward neural network. In a feed-forward neural network, every neuron only propagates an output value to a subsequent layer of the neural network 600. For example, data may only move one direction (forward) from one neuron to the next neuron in a feed-forward neural network.
In other examples, the neural network 600 is a recurrent neural network. A recurrent neural network can include one or more feedback loops, allowing data to propagate in both forward and backward through the neural network 600. This can allow for information to persist within the recurrent neural network. For example, a recurrent neural network can determine an output based at least partially on information that the recurrent neural network has seen before, giving the recurrent neural network the ability to use previous input to inform the output.
In some examples, the neural network 600 operates by receiving a vector of numbers from one layer; transforming the vector of numbers into a new vector of numbers using a matrix of numeric weights, a nonlinearity, or both; and providing the new vector of numbers to a subsequent layer of the neural network 600. Each subsequent layer of the neural network 600 can repeat this process until the neural network 600 outputs a final result at the output layer 606. For example, the neural network 600 can receive a vector of numbers as an input at the input layer 602. The neural network 600 can multiply the vector of numbers by a matrix of numeric weights to determine a weighted vector. The matrix of numeric weights can be tuned during the training of the neural network 600. The neural network 600 can transform the weighted vector using a nonlinearity, such as a sigmoid tangent or the hyperbolic tangent. In some examples, the nonlinearity can include a rectified linear unit, which can be expressed using the following equation:
y=max(x,0)
where y is the output and x is an input value from the weighted vector. The transformed output can be supplied to a subsequent layer, such as the hidden layer 604 of the neural network 600. The subsequent layer of the neural network 600 can receive the transformed output, multiply the transformed output by a matrix of numeric weights and a nonlinearity, and provide the result to yet another layer of the neural network 600. This process continues until the neural network 600 outputs a final result at the output layer 606.
Other examples of the present disclosure may include any number and combination of machine-learning models having any number and combination of characteristics. The machine-learning model(s) can be trained in a supervised, semi-supervised, or unsupervised manner, or any combination of these. The machine-learning model(s) can be implemented using a single computing device or multiple computing devices, such as the session pool 102 discussed above.
Implementing some examples of the present disclosure at least in part by using machine-learning models can reduce the total number of processing iterations, time, memory, electrical power, or any combination of these consumed by a computing device when analyzing data. For example, a neural network may more readily identify patterns in data than other approaches. This may enable the neural network to analyze the data using fewer processing cycles and less memory than other approaches, while obtaining a similar or greater level of accuracy.
Some machine-learning approaches may be more efficiently and speedily executed and processed with machine-learning specific processors (e.g., not a generic CPU). Such processors may also provide an energy savings when compared to generic CPUs. For example, some of these processors can include a graphical processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an artificial intelligence (AI) accelerator, a neural computing core, a neural computing engine, a neural processing unit, a purpose-built chip architecture for deep learning, and/or some other machine-learning specific processor that implements a machine learning approach or one or more neural networks using semiconductor (e.g., silicon (Si), gallium arsenide (GaAs)) devices. Furthermore, these processors may also be employed in heterogeneous computing architectures with a number of and a variety of different types of cores, engines, nodes, and/or layers to achieve various energy efficiencies, chip-level thermal processing considerations, processing speed improvements, data communication speed improvements, and/or data efficiency targets and improvements throughout various parts of the system when compared to a homogeneous computing architecture that employs CPUs for general purpose computing.
Often when designing and conducting experiments, factors are intended to be crossed with one another in a full factorial design. That is, every level of one factor is paired with every level of another factor, so that all possible combinations appear in the design. This may only partially happen in fractional-factorial or even optimal designs. To illustrate, consider a design for a measurement systems analysis where 3 operators are tasked with measuring 10 parts with 2 instruments. In a fully crossed design, each operator would measure every part with every instrument, repeatedly if the design is replicated multiple times. If the design is not a full factorial, perhaps due to economic constraints on the number of runs required, it could still be considered crossed if each operator had the complete combination of parts and instruments available to pull from when generating the design even if the final design does not have every combination.
In practice, circumstances such as physical constraints may arise that prevent a crossed design from being implemented. For example, suppose an experiment was to be conducted across 2 different sites, a long distance from each other. Attempting to conduct a fully crossed design would entail transporting operators, parts, and instruments from one site to the other. The goals of the experiment may be such that an alternative approach would be to have an equivalent number of operators, parts, and instruments conduct measurements at the other site, resulting in a total of 6 operators (3 at each site), 20 parts (10 at each site) and 4 instruments (2 at each site). This would be an example of a design in which some factors are nested within others, where nesting means that some of the levels of one or more factors can only be paired with one or more levels of another factor, but not all levels. In this example, the operator, part, and instrument factors are nested within the site factor since not every site can be paired with every other operator, part, and instrument.
A nesting design can refer, for example, to a design structure or design cases for a design that has at least one factor nested within another factor. Nesting designs can include some crossed factors. Nested designs are prevalent in measurement systems analysis, though they can occur in other fields as well. For instance, operators conducting an experiment may be nested within a particular lab in which they will conduct an experiment. One or more embodiments provide improvements for generating a nesting design and assessing that nesting design (e.g., coverage in the design or variability from different factors). Embodiments can avoid assigning options in particular design cases accounting for the nesting goals and can assess designs accounting for those intentional deviations from a crossed design.
The computing device 702 has a computer-readable medium 710 and a processor 708. Computer-readable medium 710 is an electronic holding place or storage for information so the information can be accessed by processor 708. Computer-readable medium 710 can include, but is not limited to, any type of random access memory (RAM), any type of read only memory (ROM), any type of flash memory, etc. such as magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips), optical disks (e.g., compact disc (CD), digital versatile disc (DVD)), smart cards, flash memory devices, etc. For instance, computer-readable medium 710 can store information obtained by the computing device 702 (e.g., received from an input device in the system 700 such as one implementing a graphical user interface 730). For example, computer-readable medium 710 obtains a first factor 712 for a design (e.g., a nesting design) and first set of candidate options 714 for the first factor. The computer-readable medium 710 also obtains a second factor 716 for a design and a second set of candidate options 718 for the second factor. For example, if the experiment is an experiment to test take-off speed for different airplanes that take different fuel types, the first factor could be the airplane used with different options being the different airplanes and the second factor could be fuel with different options indicating possible fuel types for the different airplanes. Two factors are given merely for example. Additional factors could be nested or crossed with first factor 712 and/or second factor 716 (e.g., a Factor N 724). Additional factors could also have their own candidate options (e.g., Candidate options 726 for Factor N 724). The computing device 702 could obtain additional factors and options for storage in computer-readable medium 710. Further, each set of candidate options could comprise only one option or multiple options. For instance, an additional factor could be the volume of fuel used which could take on a range of options within the tank size.
The computer-readable medium 710 could obtain information such as the factors from another computing system not shown or could obtain the information pertaining to a design using graphical user interface 730 (e.g., graphical user interface interfaces shown in examples described herein). For instance, the input interface 704 can receive a nesting indication 732 indicating that a factor is nested within another factor or nesting another factor (e.g., that the second factor 716 is nested within the first factor 712). For example, the fuel type factor could be nested within the airplane factor because options for the fuel type are dependent on the airplane selected. Embodiments described herein advantageously allow a user to control a design (e.g., by dictating the factors, options for those factors, and structure of a design such as crossed or nested factors).
Computing device 702 includes a processor 708 that executes instructions (e.g., stored at the computer-readable medium 710). The instructions can be carried out by a special purpose computer, logic circuits, or hardware circuits. In one or more embodiments, processor 708 is implemented in hardware and/or firmware. Processor 708 executes an instruction, meaning it performs or controls the operations called for by that instruction. The term “execution” is the process of running an application or the carrying out of the operation called for by an instruction. The instructions can be written using one or more programming languages, scripting languages, assembly languages, etc. Processor 708 in one or more embodiments can retrieve a set of instructions from a permanent memory device and copy the instructions in an executable form to a temporary memory device that is generally some form of RAM, for example. Processor 708 operably couples with components of computing device 702 (e.g., input interface 704, with output interface 706 and with computer-readable medium 710) to receive, to send, and to process information.
In one or more embodiments, computer-readable medium 710 stores instructions for execution by processor 708. One or more applications stored on computer-readable medium 710 are implemented in software (e.g., computer-readable and/or computer-executable instructions) stored in computer-readable medium 710 and accessible by processor 708 for execution of the instructions.
For example, in one or more embodiments, the computer-readable medium 710 comprises instructions for a design application 720 for generating a design such as a nested or crossed design. Additionally, or alternatively, the computer-readable medium 710 comprises instructions for an assessment application 722.
For example, the design application 720 can be used to assign particular groups of options of one factor to groups of options in another factor (e.g., assigning one group of fuel types to one airplane type, and another group of fuel types to another airplane type). The design application 720 can then select based on the assignment of groups particular options for design cases of a design. The assessment application 722 may be used to generate one or more computer-generated assessments of a design (e.g., assessment of a nesting design created by design application 720). For instance, an assessment may use a computer simulation of a design to assess the design before implementing it. The assessment application 722 could be used to generate the computer simulation and/or control the computer simulation (e.g., determining terms for a computer model for the computer simulation). The computing system can generate one or more computer-generated assessments of the nesting design based on the computer simulation according to the computer model.
The one or more applications can be integrated with other analytic tools. Merely for illustration, the applications are implemented using or integrated with one or more software tools such as JMP®, Base SAS, SAS® Enterprise Miner™, SAS/STAT®, SAS® High Performance Analytics Server, SAS® Visual Data Mining and Machine Learning, SAS® LASR™ SAS® In-Database Products, SAS® Scalable Performance Data Engine, SAS® Cloud Analytic Services, SAS/OR®, SAS/ETS®, SAS® Inventory Optimization, SAS® Inventory Optimization Workbench, SAS® Visual Analytics, SAS® Viya™, SAS In-Memory Statistics for Hadoop®, SAS® Forecast Server, and SAS/IML®.
The one or more applications stored on computer-readable medium 710 can be implemented as a Web application. For example, an application can be configured to receive hypertext transport protocol (HTTP) responses and to send HTTP requests. The HTTP responses may include web pages such as hypertext markup language (HTML) documents and linked objects generated in response to the HTTP requests. Each web page may be identified by a uniform resource locator (URL) that includes the location or address of the computing device that contains the resource to be accessed in addition to the location of the resource on that computing device. The type of file or resource depends on the Internet application protocol such as the file transfer protocol, HTTP, H.323, etc. The file accessed may be a simple text file, an image file, an audio file, a video file, an executable, a common gateway interface application, a Java applet, an extensible markup language (XML) file, or any other type of file supported by HTTP.
In one or more embodiments, fewer, different, or additional components can be incorporated into computing device 702. For instance, in one or more embodiments, there are multiple input devices or computing systems (e.g., one to obtain factors and/or options and one to obtain a nesting indication 732). In the same or different embodiments, there are multiple output devices or computing systems (e.g., one to display the graphical user interface 730 and one to receive a generated design or assessment).
As another example, the same interface supports both input interface 704 and output interface 706. For example, a touch screen provides a mechanism for user input and for presentation of output to the user. Alternatively, the input interface 704 has more than one input interface that uses the same or different interface technology. Alternatively, or additionally, the output interface 706 has more than one output interface that uses the same or different interface technology. In some embodiments the design application 720 and the assessment application 722 are combined into a single application.
In one or more embodiments, a computing system (e.g., the system 700 or computing device 702) implements a method as described herein (e.g., a method shown in
An operation 802 of method 800 includes receiving, using a graphical user interface, a nesting indication that the second factor is nested within the first factor. For instance, the graphical user interface can be used to structurally represent the design in a graphical representation. For example, the researcher may want a brood factor nesting the mice factor so that the design represents that individual mice are kept within their same brood in the experiment.
An operation 803 of method 800 includes generating the nesting design. For instance, where there are two factors the computing system can portion the first set of candidate options into at least two groups of a first set of groups (e.g., different brood families). The computing system can portion the second set of candidate options into at least two groups of a second set of groups (e.g., different family members). The first set of groups and the second set of groups are different. The computing system can generate, based on the nesting indication, an assignment of groups by the computing system for a design. The assignment of groups assigns a first group from the first set of groups for use with a second group of the second set of groups in the nesting design. For instance, family members 1-4 of brood 1 are assigned to brood 1 group and family members 1-4 of brood 2 are assigned to brood 2 group. The computing system then selects, based on the assignment of groups, options in the nesting design for the first factor from the first set of candidate options. The computing system may set up design cases and may decide the first 8 design cases will have brood 1 mice tested, and the next 8 design cases will have broad 2 mice tested (e.g., if the computing system has each family member replicated twice).
The computing system selects, based on the assignment of groups, options in the nesting design for the second factor from the second set of candidate options. For example, the options for assignment to a mice brood could be mice siblings born first, second, third and fourth. Because the options may be unique (e.g., a specific mouse of a specific age), the computing system can generate the assignment of groups such that any group from the first set of groups is only assigned one group from the second set of groups (i.e., mouse siblings are grouped and assigned to only one brood group), and generate a nesting design based on selecting options for the first factor from a first group of the first set of groups and selecting options for the second factor from a second group of the second set of groups assigned to the first group according to the assignment of groups.
An operation 804 of method 800 includes generating terms for a computer model for a computer simulation of the nesting design. An amount of terms (i.e., the number of those terms) in the computer model can be constrained by the computing system according to the nesting indication that the second factor is nested within the first factor. For instance, a crossed design may use additional terms such as interactions between factors to represent the potential behavior of factors in an experiment. By considering the nesting behavior, unnecessary terms can be excluded from the computer model (e.g., excluding terms related to interactions between a nesting and nested factors from other interaction terms). Additionally, or alternatively, different terms can be used (e.g., ones relevant to nesting factors).
An operation 805 of method 800 includes generating one or more computer-generated assessments of the nesting design based on the computer simulation of the nesting design according to the computer model. For instance, the computer-generated assessments could indicate a good coverage for the factors even though not all combinations will be tested given that the mice are separated in their broods according to the design for the experiment.
Operations described herein can be applied to more or fewer factors or applied iteratively based on changing factors to a design or improving assessments for a design. Operations described in method 800 can be repeated for additional factors.
For instance,
Operations can be repeated for additional factors. For instance, a computing system can use an operation 851 to obtain a third factor and a third set of candidate options for the third factor. In operation 852, the computing system can receive a subsequent indication. That subsequent indication could be that the third factor is nested within the second factor. Additionally, or alternatively, the subsequent indication could be a different relationship such as that the third factor is crossed with one or more of the first factor or the second factor. The computing system in an operation 853 can generate a design. For instance, if a first factor is nesting a second factor, it would be a nesting design even if a third factor is indicated as crossed with the first factor and/or second factor (e.g., the third factor is crossed with a first factor then nested with a second factor or the third factor is nested with a first factor, then crossed with the second factor). The computing system can generate the nesting design in which selected options for the second factor from respective candidate options are constrained according to the assignment of groups, but selected options for the third factor from respective candidate options are distributed in design cases amongst factors in which the third factor is crossed (e.g., evenly in design cases amongst first factor and second factor options).
The computing system in an operation 854 can determine terms for a computer model for a computer simulation of the nesting design. An amount of terms (i.e., the number of those terms) in the computer model can be constrained by the computing system according to an initial nesting indication and/or subsequent indications. By increasing factors, the total terms proposed by the computing system for the computer model may increase but may still be constrained to be less than all candidate terms (e.g., for a design for all crossed factors). The computing system in an operation 855 can generate one or more computer-generated assessments of the nesting design based on the computer simulation of the nesting design according to the computer model.
Example factors for the list of factors 920 could include a site factor for where a measurement in the measurement system is to occur. Additionally, or alternatively, the list of factors 920 could include an operator factor indicating multiple operator characteristics in an operator set, such as number of operators and different types of operators performing measurements. Additionally, or alternatively, the list of factors 920 could include an instrument or gauge factor indicating measurement tools of a tool set (e.g., number of measurement tools, different types, or calibrations of measurement tools). Additionally, or alternatively, the factors could include a part factor indicating a number or type of industrial product for analysis. For example, if the measurement system analysis is for measuring a rotating saw, factors in the measurement system analysis could include a part factor for a measured blade in the rotating saw, an operator factor for the operators measuring the saw blade, and a gauge factor for the calibration of the ruler for measuring the saw blade. Factors relevant to other purposes or used in other design analysis platforms could be used. These factors relevant to the measurement system analysis can be added to the factor list 920 in
Once factors are added, in
Suppose now that, in the experiment, the operators will be nested within instrument (i.e., each operator will not use every instrument), which is still nested within site along with part. In
Pop-up menu 950, or other menus, can have additional options for controlling the nesting design. For instance, an “Adjust Nesting” option may be used to adjust the ratio of options assigned to the different groups or to control the assignment of particular design cases. For instance, each time the user adds a factor to a design structure, updates can be made to keep track of each factor's relationship to other factors. Specifically, for each factor, there can be a list of other factors that directly nest it (can be empty) and another list of other factors which are nested within it (also can be empty). Additionally, or alternatively, these lists could be replaced with a global associative array list that can keep track of the entire structure. Accordingly, design cases could be interactively and dynamically assigned as the user is making structural decisions. Alternatively, relationship information could be collected when the user selects a control 980 to make a design. Keeping updated information regarding nesting relationships can be useful to ensure an appropriate model is used, for example, in diagnostics and response simulation.
Graphical user interfaces described herein can be interactive and connected. For instance, if a user changes the name of a factor in the factor list of the graphical user interface 900 in
A nesting design such as the one represented by the design structure 1010 can be useful to determine what table scripts are appropriate for follow-up analysis when a data table is created for the design (e.g., selecting option 1028 to make a table in accordance with the design 1020, so that the design-analysis process is seamless).
The design structure 1010 can also be used to determine the order in which factors appear in the design or table, so that factors near the “top” of the structure appear further to the left in the design or design table than factors closer to the “bottom” of the structure. For instance, in the design 1020, site column 1022 is shown in the left most column of factors. In contrast Part[Operator, Site] column 1024 is shown in the right most column of factors. A replicates column 1026 can show how often particular design cases are repeated within a design.
An interactive graphical user interface is shown merely as an example. Additionally, or alternatively, to interactively specifying a design structure in a graphical user interface or performing other diagnostics, users can also specify it via scripting. For example, a user could use the command: Nesting Structure({“Site”,{{“Instrument”,“Operator”}}∥“Part”), where brackets are used to indicate nesting according to the following scheme: {nesting factor,{nested factor(s)}} and the “∥” symbol represents crossing. This script can also include whenever a user saves the script to create the design. Graphical user interfaces are used in examples for explanation, but other techniques (such as scripting) could instead be used.
Design diagnostic 1080 shows the computing system not taking into accounting the nesting indication. As shown in portion 1074 there are significantly fewer model terms in design diagnostic 1070 than in design diagnostic 1080 because the model terms can be constrained based on the nesting information. The graphical user interface 1060 shows a difference in which terms are estimated by the computing system. Interactions involving the Site factor are absent in the design diagnostics 1070 accounting for nesting relationships in the nested design. These interactions are present in design diagnostic 1080 and could be misleading for test planners in a nesting design. In the design diagnostic 1080, there are also not terms of the nesting type even though the design was a nesting design.
In one or more embodiments, a computing system can receive user input indicating to further exclude or include one or more model terms of the computer-generated model terms. For instance, chart 1072 shows solid lines for each of the variance components 1076 to indicate according to the key 1073 their inclusion in a computer model for the design. One or more computer generated assessments for the design can be specific to particular assessed factor, or assessed interaction of crossed factors, in the nesting design rather than a wholistic assessment as shown in
Graphical user interface 1060 is interactive so that the user can adjust the design structure and have the design diagnostics update. Accordingly, the two design diagnostics shown in graphical user interface 1060 (i.e., design diagnostic 1070 and design diagnostic 1080) could be produced by a user initially representing the factors of the nesting design as crossed, so that the computing system would generate terms for the computer model by receiving, using the graphical user interface, a cross indication that factors were crossed with one another and generate a graphical user interface displaying candidate computer-generated model terms as show in design diagnostics 1080. Then the user could change the structure so that the computing system could receive the nesting indication after the cross indication and update the graphical user interface to display less total computer-generated terms than the candidate computer-generated model terms according to the cross indication (e.g., as shown in design diagnostic 1070). Additionally, or alternatively, the user could first represent a nesting design and change to include more crossed elements to increase the candidate terms available for a computer model.
In some embodiments, by being able to represent the nesting structure of the experiment, design assessments can be improved (e.g., assessments as part of measurement system analysis). Measurement system analysis platforms can be useful for evaluating industrial products, such as mechanical or electrical system components or substrates that can be prone to variations when manufacturing large quantities. Variations can be problematic if, for instance, the industrial products should be within certain tolerances such as for safety (e.g., automotive or airplane components) or simply to fit together well (e.g., toy snap together bricks). Variations may come from many sources such as the part itself or the measurement process for measuring the variation. Assessment of the design for conducting a measurement system can indicate variability caused by a factor of the measurement system analysis as shown here. Additionally, or alternatively, industrial parts or their measurement process can be classified based on their likelihood of meeting certain measurement standards given possible variations. Classifications or evaluations for industrial products can consider variations that come not only from a product itself, but from the measurement process such as the measurement tools, measurement operators, and measurement techniques. For instance, measurement tests can be designed and executed on a set of industrial parts for an Evaluating the Measurement Process (EMP) classification, which can be used to assess consistency, reproducibility, or repeatability of a measurement system. Accordingly, one or more computer-generated assessments can include a likelihood of classification into one of multiple groups for the product set according to a measurement standard (e.g., an EMP classification) or a likelihood of going beyond a threshold related to the measurement standard.
Embodiments described herein could be used in other platforms or for other assessments. For example, the platform could have more or fewer diagnostic measurements such as additional diagnostics for fixed effects, which can also be nested. Additionally, or alternatively, a measurement system analysis (MSA) platform could operate differently than the one in this example. For instance, in this MSA platform, multiple variances for the factors are computer-generated and distributed for each of the factors in simulations for determining the computer-generated assessments shown here. In other measurement system analysis, variance can be supplied by the user for simulating the factors according to a user supplied variance such as the platform shown and described in U.S. Pat. No. 11,346,751, issued May 31, 2022, and incorporated by reference.
For instance,
In the design diagnostics portion 1130 of the graphical user interface 1100, a user can change aspects of the design 1110 to affect one or more computer-generated assessments for the design 1110. For instance, a computing system can receive one or more change indications from the user adjusting an amount or range of candidate options for a given factor of a design (e.g., a number of levels for factors). Additionally, or alternatively, the computing system can receive one or more change indications from the user adjusting an amount (i.e., number) of replicates of design cases in a design. Additionally, or alternatively, the computing system can receive one or more change indications from the user adjusting a number of “runs” or design cases in a design. Additionally, or alternatively, the user can select factor column 1132 to increase or decrease factors in the design (e.g., from an earlier provided factor list). The computing system can update the graphical user interface to display at least one updated computer-generated assessment accounting for the change indication (e.g., in the below variance component estimator performance section 1136 of the graphical user interface 1100). Accordingly, a user can engage in “what if” analysis by looking at different changes that could be made to the design to see how it would affect computer-generated assessments. For instance, the user can use a computing system to explore questions like what if the number of levels for a factor is 2 or what if it is 5? The computing system can show how these differences would affect the evaluation of the resulting design. In some embodiments, a computing system can make these adjustments without updating the particular design cases displayed in design 1110 (e.g., using the design diagnostic portion 1130).
Guided prompts such as in
One or more embodiments described with respect to categorical factors are also applicable to continuous factors.
In embodiments, such as the example in
For example, an operation 1603 of method 1600 in
The computing system can use the master array 1790 to re-sort factors in hierarchical order (including a re-sort of the master array itself and its entries). For instance, the organization of the names vector 1792 can be changed with corresponding changes in the internal array 1794. A master array (e.g., the master array 1790) can be used to create a model, which can serve as the basis for design, evaluation, and simulation. For instance, an operation 1604 of method 1600 includes creating a design. The computing system can use one or more computing structures created (e.g., in operations 1601-1603) to portion candidate options for the factors in a design. An operation 1605 of method 1600 includes creating diagnostics. For instance, the computing design can receive a nesting indication by receiving a level in a nesting hierarchy for each factor of the nesting design (e.g., based on hierarchical order in a master array), and generate the one or more computer-generated assessments based on the nesting hierarchy (e.g., based on simulations for a resulting design according to the hierarchical order).
Graphical user interface 1820 in
Claims
1. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including system instructions operable to cause a computing system to:
- obtain a first factor for a nesting design and a first set of candidate options for the first factor;
- obtain a second factor for the nesting design and a second set of candidate options for the second factor;
- display, in an interactive graphical user interface, first and second user controls representing the first and second factors, respectively;
- receive, using the interactive graphical user interface, user input arranging the second user control to be nested within the first user control;
- receive, using the interactive graphical user interface and responsive to the arranging, a nesting indication that the second factor is nested within the first factor;
- responsive to receiving the nesting indication, generate an adjustable design structure representing the nesting design and one or more cross-relationships between factors in the nesting design by: portioning, by the computing system, the first set of candidate options into at least two groups of a first set of groups; portioning, by the computing system, the second set of candidate options into at least two groups of a second set of groups, wherein the first set of groups and the second set of groups are different; generating, based on the nesting indication, an assignment of groups by the computing system for a design, wherein the assignment of groups assigns a first group from the first set of groups for use with a second group of the second set of groups in the nesting design; selecting, by the computing system, based on the assignment of groups, options in the nesting design for the first factor from the first set of candidate options; and selecting, by the computing system, based on the assignment of groups, options in the nesting design for the second factor from the second set of candidate options; and wherein the adjustable design structure is configured to determine an order in which the first and second factors appear in the nesting design;
- generate a computer simulation of the nesting design, wherein to generate the computer simulation, the system instructions are operable to cause the computing system to generate terms for a computer model for the computer simulation of the nesting design, and wherein an amount of terms in the computer model are constrained by the computing system according to the nesting indication that the second factor is nested within the first factor;
- execute the computer simulation of the nesting design according to the terms generated for the computer model;
- receive one or more responses output by the computer simulation; and
- generate one or more computer-generated assessments of the nesting design based on the one or more responses output by the computer simulation of the nesting design according to the computer model, wherein the one or more responses output by the computer simulation are input parameters for generating the one or more computer-generated assessments.
2. The computer-program product of claim 1, wherein the system instructions are operable to cause the computing system to:
- generate the assignment of groups such that any group from the first set of groups is only assigned one group from the second set of groups; and
- generate the nesting design based on selecting options for the first factor from a first group of the first set of groups and selecting options for the second factor from a second group of the second set of groups assigned to the first group according to the assignment of groups.
3. The computer-program product of claim 1, wherein the system instructions are operable to cause the computing system to:
- select options for the first factor and the second factor as input into the computer simulation simulating one or more responses according to the nesting design; and
- receive the one or more responses output by the computer simulation based on the input.
4. The computer-program product of claim 1, wherein the system instructions are operable to cause the computing system to:
- display, in the graphical user interface, the one or more computer-generated assessments;
- receive using the graphical user interface a change indication to change one or more of: an amount of factors in the nesting design; an amount or range of candidate options for a given factor of the nesting design; and an amount of replicates of design cases in the nesting design; and
- update the graphical user interface to display at least one updated computer-generated assessment accounting for the change indication.
5. (canceled)
6. The computer-program product of claim 1, wherein the system instructions are operable to cause the computing system to:
- obtain a third factor and a third set of candidate options for the third factor;
- receive, using the graphical user interface, a cross indication that the third factor is crossed with one or more of the first factor or the second factor; and
- generate the nesting design in which: selected options for the third factor from the third set of candidate options are distributed in design cases amongst factors in which the third factor is crossed; and selected options for the second factor from the second set of candidate options are constrained according to the assignment of groups.
7. The computer-program product of claim 1, wherein the system instructions are operable to cause the computing system to:
- obtain a third factor and a third set of candidate options for the third factor;
- receive a subsequent indication that the third factor is nested within the second factor; and
- wherein the amount of terms in the computer model are constrained by the computing system according to the nesting indication and the subsequent indication.
8. The computer-program product of claim 1, wherein the system instructions are operable to cause the computing system to:
- receive the nesting indication by displaying in the graphical user interface a guided prompt for user input indicating the second factor is nested within the first factor; and
- generate the one or more computer-generated assessments by generating computing instructions for the one or more computer-generated assessments based on the guided prompt.
9. The computer-program product of claim 1, wherein the system instructions are operable to cause the computing system to:
- receive the nesting indication by receiving a level in a nesting hierarchy for each factor of the nesting design; and
- generate the one or more computer-generated assessments based on the nesting hierarchy.
10. The computer-program product of claim 1, wherein:
- the nesting indication further indicates to cross a third factor with both the second factor and the first factor; or
- the nesting indication further indicates to cross a third factor with the first factor, then nest the second factor within the first factor.
11. The computer-program product of claim 1, wherein the nesting indication further indicates to fully nest a series of multiple factors in a design.
12. The computer-program product of claim 1, wherein the system instructions are operable to cause the computing system to:
- generate, in response to the nesting indication, a computing structure indicating nesting and cross relationships between factors in the nesting design, wherein the computing structure comprises one or more of a computing vector, computing list and a computing array; and
- portion the first set of candidate options and the second set of candidate options based on the computing structure.
13. The computer-program product of claim 1, wherein the system instructions are operable to cause the computing system to:
- generate the one or more computer-generated assessments of the nesting design by generating computer-generated assessments for assessed factors, or assessed interaction of crossed factors, in the nesting design; and
- display, in the graphical user interface, an indication of nesting relationships for the assessed factors, or the assessed interaction of crossed factors.
14. The computer-program product of claim 1, wherein the computing system is configured to generate terms for the computer model by determining one or more model types based on one or more factor statuses for the first factor indicating one or more of:
- the first factor is nesting another factor of the nesting design,
- the first factor is nested within another factor of the nesting design, and
- the first factor is crossed with another factor of the nesting design.
15. The computer-program product of claim 1,
- wherein the first factor is a categorical factor, and the first set of candidate options comprise category options for the first factor; and
- wherein the second factor is a continuous factor, and the second set of candidate options comprises a range set of one or more ranges for options for the second factor; and
- wherein the portioning the first set of candidate options comprises distributing the category options to each group of the first set of groups; and
- wherein the portioning the second set of candidate options comprises distributing one or more ranges of the range set to each group of the second set of groups.
16. The computer-program product of claim 1, wherein the system instructions are operable to cause the computing system to:
- generate multiple candidate designs;
- generate the nesting design by selecting from the multiple candidate designs based on the one or more computer-generated assessments; and
- display a graphical representation of the nesting design in the graphical user interface, wherein the graphical user interface indicates nesting relationships for factors in the nesting design.
17. The computer-program product of claim 1, wherein the system instructions are operable to cause the computing system to:
- obtain the first factor and the second factor pertaining for a design for a measurement system analysis, wherein the measurement system analysis comprises measurement tests for evaluating a product set comprising one or more industrial products; and
- generate the one or more computer-generated assessments indicating one or more of: a variability caused by a factor of the measurement system analysis; a likelihood of classification into one of multiple groups for the product set according to a measurement standard; and a likelihood of going beyond a threshold related to the measurement standard.
18. The computer-program product of claim 1, wherein the computing system is configured to generate terms for the computer model by:
- displaying, in the graphical user interface, computer-generated model terms indicating terms of a nested type; and
- receiving user input indicating to further exclude at least one model term of the computer-generated model terms.
19. The computer-program product of claim 18, wherein the computing system is configured to generate terms for the computer model by displaying, in the graphical user interface, the computer-generated model terms further indicating terms of an interaction type for crossed factors of the nesting design.
20. The computer-program product of claim 1, wherein the computing system is configured to generate terms for the computer model by:
- receiving, using the graphical user interface, a cross indication that the second factor is crossed with the first factor;
- generating an additional graphical user interface displaying candidate computer-generated model terms;
- receiving the nesting indication subsequent to the cross indication; and
- updating the additional graphical user interface to display less total computer-generated terms than the candidate computer-generated model terms.
21. A computer-implemented method comprising:
- obtaining a first factor for a nesting design and a first set of candidate options for the first factor;
- obtaining a second factor for the nesting design and a second set of candidate options for the second factor;
- displaying, in an interactive graphical user interface, first and second user controls representing the first and second factors, respectively;
- receiving, using the interactive graphical user interface, user input arranging the second user control to be nested within the first user control;
- receiving, using the interactive graphical user interface and responsive to the arranging, a nesting indication that the second factor is nested within the first factor;
- responsive to receiving the nesting indication, generating an adjustable design structure representing the nesting design and one or more cross-relationships between factors in the nesting design by: portioning, by a computing system, the first set of candidate options into at least two groups of a first set of groups; portioning, by the computing system, the second set of candidate options into at least two groups of a second set of groups, wherein the first set of groups and the second set of groups are different; generating, based on the nesting indication, an assignment of groups by the computing system for a design, wherein the assignment of groups assigns a first group from the first set of groups for use with a second group of the second set of groups in the nesting design; selecting, by the computing system, based on the assignment of groups, options in the nesting design for the first factor from the first set of candidate options; and selecting, by the computing system, based on the assignment of groups, options in the nesting design for the second factor from the second set of candidate options; and wherein the adjustable design structure is configured to determine an order in which the first and second factors appear in the nesting design;
- generating a computer simulation of the nesting design by generating terms for a computer model for the computer simulation of the nesting design, and wherein an amount of terms in the computer model are constrained by the computing system according to the nesting indication that the second factor is nested within the first factor;
- executing the computer simulation of the nesting design according to the terms generated for the computer model;
- receiving one or more responses output by the computer simulation; and
- generating one or more computer-generated assessments of the nesting design based on the one or more responses output by the computer simulation of the nesting design according to the computer model, wherein the one or more responses output by the computer simulation are input parameters for generating the one or more computer-generated assessments.
22. The computer-implemented method of claim 21,
- wherein the generating the assignment of groups comprises generating the assignment of groups such that any group from the first set of groups is only assigned one group from the second set of groups; and
- wherein the generating the nesting design comprises generating the nesting design based on selecting options for the first factor from a first group of the first set of groups and selecting options for the second factor from a second group of the second set of groups assigned to the first group according to the assignment of groups.
23. The computer-implemented method of claim 21,
- wherein the computer-implemented method further comprises: selecting options for the first factor and the second factor as input into the computer simulation simulating the one or more responses according to the nesting design; and receiving the one or more responses output by the computer simulation based on the input.
24. The computer-implemented method of claim 21, wherein the computer-implemented method further comprises:
- displaying, in the graphical user interface, the one or more computer-generated assessments;
- receiving using the graphical user interface a change indication to change one or more of: an amount of factors in the nesting design; an amount or range of candidate options for a given factor of the nesting design; and an amount of replicates of design cases in the nesting design; and
- updating the graphical user interface to display at least one updated computer-generated assessment accounting for the change indication.
25. (canceled)
26. The computer-implemented method of claim 21,
- wherein the computer-implemented method further comprises: obtaining a third factor and a third set of candidate options for the third factor; and receiving, using the graphical user interface, a cross indication that the third factor is crossed with one or more of the first factor or the second factor; and
- wherein the generating the nesting design comprises generating the nesting design in which: selected options for the third factor from the third set of candidate options are distributed in design cases amongst factors in which the third factor is crossed; and selected options for the second factor from the second set of candidate options are constrained according to the assignment of groups.
27. The computer-implemented method of claim 21,
- wherein the computer-implemented method further comprises: obtaining a third factor and a third set of candidate options for the third factor; and receiving a subsequent indication that the third factor is nested within the second factor; and
- wherein the amount of terms in the computer model are constrained by the computing system according to the nesting indication and the subsequent indication.
28. The computer-implemented method of claim 21,
- wherein the receiving the nesting indication comprises displaying in the graphical user interface a guided prompt for user input indicating the second factor is nested within the first factor; and
- wherein the generating the one or more computer-generated assessments comprises generating computing instructions for the one or more computer-generated assessments based on the guided prompt.
29. The computer-implemented method of claim 21, wherein the generating terms for the computer model comprises:
- displaying, in the graphical user interface, computer-generated model terms indicating terms of a nested type; and
- receiving user input indicating to further exclude at least one model term of the computer-generated model terms.
30. A computing device comprising processor and memory, the memory containing instructions executable by the processor wherein the computing device is configured to:
- obtain a first factor for a nesting design and a first set of candidate options for the first factor;
- obtain a second factor for the nesting design and a second set of candidate options for the second factor;
- display, in an interactive graphical user interface, first and second user controls representing the first and second factors, respectively;
- receive, using the interactive graphical user interface, user input arranging the second user control to be nested within the first user control;
- receive, using the interactive graphical user interface and responsive to the arranging, a nesting indication that the second factor is nested within the first factor;
- responsive to receiving the nesting indication, generate an adjustable design structure representing the nesting design and one or more cross-relationships between factors in the nesting design by: portioning, by the computing device, the first set of candidate options into at least two groups of a first set of groups; portioning, by the computing device, the second set of candidate options into at least two groups of a second set of groups, wherein the first set of groups and the second set of groups are different; generating, based on the nesting indication, an assignment of groups by the computing device for a design, wherein the assignment of groups assigns a first group from the first set of groups for use with a second group of the second set of groups in the nesting design; selecting, by the computing device, based on the assignment of groups, options in the nesting design for the first factor from the first set of candidate options; and selecting, by the computing device, based on the assignment of groups, options in the nesting design for the second factor from the second set of candidate options; and wherein the adjustable design structure is configured to determine an order in which the first and second factors appear in the nesting design;
- generate a computer simulation of the nesting design, wherein to generate the computer simulation, the system instructions are operable to cause the computing system to generate terms for a computer model for the computer simulation of the nesting design, and wherein an amount of terms in the computer model are constrained by the computing device according to the nesting indication that the second factor is nested within the first factor;
- execute the computer simulation of the nesting design according to the terms generated for the computer model;
- receive one or more responses output by the computer simulation; and
- generate one or more computer-generated assessments of the nesting design based on the one or more responses output by the computer simulation of the nesting design according to the computer model, wherein the one or more responses output by the computer simulation are input parameters for generating the one or more computer-generated assessments.
Type: Application
Filed: Aug 4, 2022
Publication Date: Aug 31, 2023
Inventors: Caleb Bridges King (Morrisville, NC), Joseph Albert Morgan (Raleigh, NC), Ryan Adam Lekivetz (Cary, NC), Bradley Allen Jones (Cary, NC), Jacob Davis Rhyne (Dallas, NC)
Application Number: 17/881,120
