Method and system for generating an assignment binder within an assessment management system
A system of hardware and software provides a systematic and systemic method for performing assessment of the effectiveness of different levels within an institution. The system includes template driven definitions that span multiple tiers of hierarchically arranged operating units within the institution thereby providing flexibility and customization to handle a user-definable model of the institution's organization. In particular, an assignment binder may be created based on selective filtering of artifacts. Filtering criteria may be used to select, for inclusion within the assignment binder, only those artifacts from authors having certain attributes.
The present application claims priority to U.S. Provisional Patent Application Ser. No. 60/670,963, filed Apr. 12, 2005, entitled “Transaction, Commerce, Community, Content, Learning, and Evaluation Systems”, the disclosure of which is incorporated herein by reference in its entirety. The present application is a continuation of U.S. patent application Ser. No. 11/363,868, filed Feb. 27, 2006, entitled “Method and System for Assessment Within a Multi-Level Organization”, the disclosure of which is incorporated herein in its entirety.
BACKGROUND1. Field
The present disclosure relates generally to computer software and hardware systems, and more particularly, to such a system for providing assessment of multiple levels of effectiveness.
2. Background
As part of providing quality educational opportunities, many academic institutions are interested in assessing and evaluating their performance and efforts as well as the performance of units and individuals within the institution.. These institutions can include, for example, on-line institutions, K-12 schools, colleges, universities and even multi-campus university systems. One purpose of assessment involves collecting and documenting the necessary evidence needed for satisfying the requirements of regional or national accreditation bodies. Another purpose is to allow the institution to make informed and objective decisions about, for example, curriculum content, course offerings, and educational effectiveness.
In the past, assessment efforts have been disjointed and performed in relative isolation at the different levels of the academic institution. For example, the assessment process at the classroom level was performed separate from any assessment (if ever performed) at a department level or higher (e.g., institutional level). This approach lead to redundancy of information and efforts, as well as disparate results that would need to be combined in some manner and evaluated if an overall assessment result is desired.
Furthermore, identifying and collecting information needed to meet accreditation requirements may be performed on an ad-hoc basis when needed. These processes are then repeated the next time accreditation is reviewed. Such an approach relies heavily on repeatedly reviewing the same information and generating the necessary documentation each time accreditation is reviewed.
Accordingly, there exists a need for an assessment system for institutions that is able to consider and connect multiple organizational tiers of the institution and to provide systematic and systemic data collection, organization, and retrieval components to support institutional assessment efforts.
SUMMARYAccordingly, one aspect of the present invention relates to a method for selectively assembling an assignment binder including a selected subset of a plurality of artifacts. In accordance with this method, first input is received defining a criteria, the criteria related to a plurality of identifying attributes associated, respectively, with each of the plurality of artifacts. A selected subset of artifacts is identified based on those of the plurality of artifacts having associated identifying attributes matching the criteria, wherein the criteria relates to at least two criterion related to one or more of demographic attributes, course-related attributes, operating unit-related attributes, section-related attributes, goal-related attributes, and educational experience-related attributes. The selected subset of artifacts are collected within the assignment binder.
Another aspect of the present invention relates to a method for selectively assembling an assignment binder, for a multi-level institution. In accordance with this method, first input is received defining a criteria, the criteria related to a plurality of identifying attributes associated, respectively, with each of the plurality of artifacts and the criteria are forwarded to a separate computer-based system of the institution, wherein the separate computer-based system is configured to store the plurality of artifacts. In a response from the separate computer-based system, the selected subset of those of the plurality of artifacts having associated identifying attributes matching the criteria are identified.
Yet another aspect of the present invention relates to a system for selectively assembling an assignment binder including a selected subset of a plurality of artifacts. This system includes a programmable computer that is configured to a) receive first input defining a criteria, the criteria related to a plurality of identifying attributes associated, respectively, with each of the plurality of artifacts; b) identify the selected subset based on those of the plurality of artifacts having associated identifying attributes matching the criteria, wherein the criteria relates to at least two criterion related to one or more of demographic attributes, course-related attributes, operating unit-related attributes, section-related attributes, goal-related attributes, and educational experience-related attributes; and c) collect the selected subset of artifacts within the assignment binder.
Still another aspect of the present invention relates to a system for selectively assembling an assignment binder, for a multi-level institution, including a selected subset of a plurality of artifacts. The system includes a programmable computer that is configured to a) receive first input defining a criteria, the criteria related to a plurality of identifying attributes associated, respectively, with each of the plurality of artifacts; b) forward the criteria to a separate computer-based system of the institution, wherein the separate computer-based system is configured to store the plurality of artifacts; and c) receive from the separate computer-based system the selected subset of those of the plurality of artifacts having associated identifying attributes matching the criteria.
It is understood that other embodiments of the present invention will become readily apparent to those skilled in the art from the following detailed description, wherein it is shown and described only various embodiments of the invention by way of illustration. As will be realized, the invention is capable of other and different embodiments and its several details are capable of modification in various other respects, all without departing from the spirit and scope of the present invention. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.
BRIEF DESCRIPTION OF THE DRAWINGSVarious aspects of an institutional assessment system are illustrated by way of example, and not by way of limitation, in the accompanying drawings, wherein:
The figures include a number of different screen shots of an exemplary user interface. One of ordinary skill will appreciate that the depicted interface is provided by way of example and that the layout, options and items on each screen shot may be altered or augmented without departing from the scope of the present invention.
DETAILED DESCRIPTIONThe detailed description set forth below in connection with the appended drawings is intended as a description of various embodiments of the invention and is not intended to represent the only embodiments in which the invention may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without these specific details. In some instances, well known structures and components are shown in block diagram form in order to avoid obscuring the concepts of the invention. Additionally, the term “automatic” may be used herein to describe one or more process steps that may be performed in an automated manner using various types of programmable processors or computers. However, one of ordinary skill will recognize that the performance of these steps may also be accomplished manually or via a combination of manual and automated processes.
Terminology
In general, as used herein, “assessment” means a repeating or cyclic improvement process that involves measuring outcomes versus stated objectives or goals and identifying corrective action. The term “evaluation” has more of a connotation of a point in time measurement of one or more outcomes versus stated objectives without necessarily having a built-in improvement step. For example, a program may have the goal of developing students' critical thinking abilities and might annually determine whether, as a program, that goal has been achieved. Therefore, one might wish to annually assess this by looking at the aggregate performance of the program's students on a senior capstone project.
Other terms that are used consistently herein include:
Mission: A broad statement that describes the over-arching purpose of an organization. Mission statements typically are not measurable because of the scope that they encompass and because they are not time-constrained.
Goal: Missions are frequently broken down further into a series of goals. Though more specific than a mission, goals are still broad statements and may not be easily measurable. Goals provide guidance on areas that should be addressed through specific, measurable objectives.
Objective: An objective is an expression of the intended result or consequence of some activity (e.g. instruction or some other performance). It differs from an outcome, which is the achieved result, although the two are sometimes used interchangeably. The activities described by an objective should be measurable within a defined timeframe. Frequently, the term is used with a modifier to clarify the activity, e.g. a learning objective or a performance objective.
Standard: In educational assessment, for example, a standard is a statement of what a learner should know or be able to do. A standard is more specific than a goal and should be measurable. In that sense, a standard is much closer to an objective than a goal. However, unlike an objective, a standard may not be related to a specific activity, instruction, performance, etc. or timeframe. Also, standards are frequently generated by organizations external to the academic institution (e.g. state governments, accrediting agencies) as a means to standardize measurement of performance or learning across multiple organizations.
Outcome: An outcome is the achieved result or consequence of some activity (e.g. instruction or some other performance). It differs from an objective, which is the intended result, although the two are sometimes used interchangeably. Frequently, the term is used with a modifier to clarify the activity, e.g. a learning outcome or a performance outcome.
A “template” is a means of defining the structure of various data in the system and the instruments used to collect that data. A template may define one or more customizable forms and the attributes that are present on each form. A “form” is a customizable page within a template consisting of one or more attributes. An “attribute” is a piece of data that defines an entity (such as the name of a user) which may be represented on a form as a field. Certain attribute fields may be required by the system. The level of customization (e.g. the ability to change labels on an attribute versus the ability to add new attributes to a form) may be dependent on the type of template that the form is a part of.
For example, a template for collecting data about a user might consist of one form. On that form are specific attributes, such as “name” “social security number” and “type.” The “name” and “social security number” attributes might allow for entry of text information, while the “type” attribute might offer a selection of ‘faculty,’ ‘staff,’ or ‘student.’ The “name” and “social security number” attributes might be required, meaning that any person completing the form must enter data for those attributes, while the “type” attribute might be optional. One of ordinary skill will recognize that there are numerous different and varying attributes that can be defined for each form using standard available web-based interfaces. For example, the information related to an operating unit will vary for a “College” operating unit and a “Department” operating unit. A College operating unit may have fields to designate Dean, while a Department would have fields to identify the current Chair of the department. Beyond such labeling distinctions, different operating units may even have different types of data. For example, an operating unit representing a Center for Media Students may need to store, as part of the operating unit, information about the grant that funds the Center, the grant's initiation and expiration dates, and who manages the Grant, while a Department, funded out of the university budget, may only need to store university budget codes assigned to the Department. Based on their purpose and level within the organization, different operating units will have different informational needs.
Templates may also consist of more than one form; this is particularly useful for the representation of a data collection process that consists of multiple steps. For example, an Assessment Project template might consist of a “properties form” to collect data about the properties of the project itself, one or more “objective forms” to collect data about each objective being measured, and four “activity forms” per objective to collect data about the different steps of assessing the performance of the objective. Each form would have specific attributes defined. This exemplar template is illustrated in
While many examples are provided herein that specifically include a higher-education institution, the principles of the present invention contemplate other types of institutions as well. For example, corporations, governmental entities, and K-12 institutions are all considered within the scope of the present invention. An institution may also be a consortium of schools and/or campuses. In general terms, an institution is an operating unit and is, itself, made up of different operating units that may correspond to campuses, colleges, departments, sub-departments, etc. The systems and methods described herein do not require any particular arrangement of operating units but, instead, allow the institution to model its organization into a hierarchy of operating units for purposes of management, planning, and reporting of assessment efforts. The terms “unit”, “operating unit”, and “operational unit” are often used interchangeably herein. However, there are unique aspects of operating units related to their respective level in the organizational hierarchy that allow customization and flexibility based on these unique aspects.
In the higher education example, operating units offer programs of study, or simply “programs”, with different courses and sections. Thus, assessment can occur, for example, at or across an institutional level, a departmental level, a program level, an individual level, a course level and at the section level. Programs may also include courses offered by different operating units. For example, a particular program of study may require both an introductory Engineering course and a computer science course. Within the institutional level assessment, different varieties of assessment may also be performed on different levels of operating units as well. For example, the institution may be organized according to operating units that represent department levels, college levels, etc. Thus, a framework is described herein, in which a hierarchically arranged group of organizational units and activities are defined and assessed against various criteria and objectives.
The applications 108 may also access data storage facilities 112 and other computer systems 114. For example, the data facilities 112 may be one or more databases having assessment project data stored and arranged in a convenient and appropriate manner for easy manipulation and retrieval. The other computer systems 114 may be a variety of third-party systems that contain data or resources that are useful for the assessment system 102. In the exemplary higher education environment, the systems 114 may include a student information system (SIS) that maintains student demographic information as would be appreciated by one of ordinary skill. The systems 114 may also include an electronically maintained class, or course, schedule for the institution that includes information about the courses such as section numbers, professors, class size, department, college, the students enrolled, etc. Other campus-related systems such as financial aid and the bursar's office may be included in the systems 114 of
Another resource to which the back end 110 may provide connectivity is a campus (or institutional) academic system 116. An example of such a system is provided by the present Assignee under the name Academic Suite™ with many features thereof described in U.S. Pat. No. 6,998,138 entitled “Internet-Based Education Support System and Methods”, the disclosure of which is incorporated herein in its entirety. As described in that patent, the campus academic system 116, in an academic environment, provides a platform that allows students and teachers to interact in a virtual environment based on the courses for which the student is enrolled. This system may be logically separated into different components such as a learning system, a content system, a community system, and a transaction system. An example of such a student is the user 118 that can access the academic system 116 via a web browser or similar interface. The user 118 may also be faculty, staff or an administrative officer. An example of these separate components of the system 116 are described in detail in pending patent applications “Method and System for Conducting Online Transactions (Ser. No. 10,373,924 filed Feb. 25, 2003), “Content and Portal Systems and Associated Methods” (Ser. No. 11/142,965 filed Jun. 2, 2005), and “Content System and Associated Methods” (Ser. No. 10/918,016 filed Aug. 13, 2004), all of which are incorporated herein by reference in their entirety.
Of particular usefulness to the assessment system 102, the academic system 116 provides a virtual space that the user 118 may visit to receive information and to provide information. One exemplary arrangement provides the user 118 with a home page where general information may be located and that has links to access course-specific pages where course-specific information is located. As explained in the incorporated patent and patent applications, electronic messaging, electronic drop boxes, and executable modules may be provided within the user's virtual space on the academic system 116. Thus, with respect to the assessment system 102, one of the applications 108 may be used to generate information that is to be deployed to one or more users of the academic system 116. Via the back-end 110, the information may be sent to the academic system 116 where it is made available to the user 118 just as any other information is made available. Similarly, from within the academic system 116, the user may enter and submit data that is routed through the back end 110 to one of the applications 108. One of ordinary skill will recognize that the academic system 116 and the assessment system 102 may be more closely integrated so that the connectivity between the applications 108 and the system 116 is achieved without a network connection or special back end software 110.
Although the front end 106, applications 108, and back end 110 of the assessment system 102 are each depicted as a single block in
Assessment within an institution such as a higher-education academic institution is a complex undertaking that encompasses many different levels of evaluation, data collection, and correction. For example, at the institutional level, a university may be focused on assessing accreditation requirements and strategic planning initiatives. At the program level, the relevant managers may be focused on assessing program effectiveness and discipline-specific accreditation requirements. At the classroom level, the instructors may be interested in assessing course design and curriculum coverage. Thus, some of the diverse questions that one or more assessment processes may try to address are decisions on faculty promotion and tenure, determining an effective mix of full-time and adjunct faculty, determining an effective mix of traditional and remote courses, and assessing whether the needs of specific student populations (e.g., commuters, transferees, minorities) are being met.
Assessment Projects and Initiatives
In general the phrase “assessment project” is used herein to describe efforts of an institution, an operating unit, a program or a course to assess and evaluate different aspects of its operation and effectiveness. One or more assessment projects may be ongoing at any time for the same operating unit or different operating units. An “assessment initiative” may encompass a number of different assessment projects that are related to a common theme. For example, an assessment initiative for Accreditation Board of Engineering and Technology (ABET) accreditation may be defined by the College of Engineering such that the ABET assessment project of each of the individual departments within the College (e.g. Department of Electrical Engineering, Department of Civil Engineering, Department of Mechanical Engineering, etc.) may connect to the College-level assessment initiative to enable College-level reporting on status and outcomes. By explicitly connecting, or associating, a number of assessment projects with an assessment initiative, a user of the present system is able to aggregate and organize data from among the different projects so that assessment and analysis can occur from both an individual project perspective as well as from the assessment initiative's cross-project perspective. Exemplary assessment projects might be:
a) internal to the program, such as a curriculum planning project or a program review;
b) internal to the institution, such as fulfilling institutional strategic planning requirements; or
c) for external review or accreditation, such as completing an institutional accreditation report or a discipline-specific accreditation report.
Assessment projects may be initiated at different times and will span different time frames. For example, while a project supporting the unit's responsibilities for institutional accreditation may actually take place over several years, strategic planning assessment projects may only last eight months and happen with more regularity than accreditation.
Even though the assessment process can be highly complex, there are some general provisions of the assessment system 102 of
From a starting page, the user makes a selection, in step 202, via a user interface to enter a goal and planning component of the assessment system 102. After activating such a component, the user is prompted to define an assessment project. To simplify this process, the system 102 may provide the user with a default template or form that has some predefined fields and information. Alternatively, the user may be presented with a number of different types of templates and asked to make a selection. For example, the user may be asked whether the assessment project involves a program, a department, a college, an institution, or a course. Based on the user's response, an appropriate template may be provided to the user. For example, at the course level, the user would likely not be interested in defining strategic planning goals. Similarly, at the institutional level, the user would likely not be interested in specifying a course objective. Thus, the templates provided to the user may have their initial content dependent on what type of assessment project is being defined. In one particular embodiment, the assessment system 102 may be initially provided with a number of predefined templates. As part of the initialization process (before assessment projects are defined) a user or other administrator may customize these templates in accordance with the institution and its programs, departments, courses, etc. The customization of templates is not a necessary step before an assessment project is begun but it advantageously provides flexible customization geared towards each respective institution and their hierarchical organization.
As a result, in step 204, the user selects a template for the type of assessment project being generated. Then, in step 206, the user collects and enters data required by the forms as defined in the selected template. For example, the collection and entering of data may include metadata about the assessment project, the specific objectives of the assessment project, and activities related to each objective. Once the form is completed, then it can be presented to other users for review and approval in step 208. For example, a department chair may have responsibility for creating an assessment project but there may be a committee that may modify and approve the assessment project before it can be implemented.
In the above description, it is assumed that the user has adequate access rights within the assessment system 102 to perform the desired tasks. As known to one of ordinary skill, user identities and user roles may be employed to restrict user activities so that certain users have more access privileges to data resources than other users. These access privileges may be relatively static or may be more dynamic in nature as various users are placed in different roles or committees. Certain users may be allowed no access to an assessment project, read-only access to the project, edit-access to the assessment project, or create-access to the assessment project. Additionally, within an assessment project, the individual components of the project may have controlled access based on user roles as well. Furthermore, an assessment project manager role may be employed, and assigned to a user, to help with administering and managing the project and its implementation. Thus, different resources within the present assessment system may be protected through the application of access privileges that allow each resource to be available to one or more identified users or shared by all users.
a) a main form 252 is used to collect information about the project itself. Additional fields can be added to a project form in new templates by the user, but certain fields may be required on an assessment project template's project form in order to standardize data collection. While not absolutely necessary, it is advantageous to permit only one project form per assessment project template 250.
b) An objective form 254 is used to collect information about the objectives of the assessment project. An assessment project typically has one or more objectives, so at least one objective form is present. Many assessment projects have multiple objectives, however.
c) Objective forms can have one or more activity forms 256-262 associated with them in the template 250. These activity forms 256-262 collect information about various activities related to conducting the assessment of the objective. Each activity form may have a different set of fields on it. Addition of an objective (e.g. addition of an objective form 254) to a template 250, also generates the associated activity forms 256-262.
The template 250 of
In one example, the template 250 of
Summary Statement: This provides an area to enter a general description or summary of the whole assessment project. This summary can also be used as part of a report generated for the assessment project. For example, the summary statement could include an overview of the project, names and titles of the university faculty and staff involved, and a summary of the findings. This might appear as the first page of a report.
Start/End Dates: Projects to evaluate learning outcomes or reviews of programs at an institution may have specific time periods which they span (e.g. an accreditation project may last several years, but a program review may happen annually.) A user initiating an assessment project can specify start and end dates for the assessment project. The default start date would be the project's date of creation. The default end date would not be set (i.e. no end date) as some assessment projects could be continuous and ongoing.
Phase Summaries: For each phase that is defined for the assessment project by the template, the user (or manager) is able to enter summary information for that phase.
Recurrence: The assessment process is cyclical, so frequently the same (or similar) assessment project may be run on a regular basis. For example, a departmental program review may occur every year, or the department's discipline-specific accreditation may need to be re-confirmed every three years. It is therefore useful to track if an assessment project is a re-occurrence of some previous assessment project.
Workflow: Workflow status can be used to control the state of the template and to determine what action or activity is next in the process flow.
The flowchart of
Before beginning the definition of an assessment project, the institution itself may be modeled.
Thus, in step 270, initiation of an assessment project may begin by modeling the hierarchical structure of the institution. Once this is accomplished, definition of an assessment project may begin, in step 272. The assessment project within an operating unit (e.g., a college, a department) may be related to a higher-level operating unit for which goals and objectives have already been defined. In an educational institution, for example, the sub-units may be academic departments, administrative or other departments. Underneath these units may be courses or course sections. When creating other assessment projects, the previous modeling of the institution (step 270) does not need to be re-accomplished but may be re-utilized.
For the sub-operating units, their own objectives are defined in step 274. These objectives may be aligned with the top-level goals and objectives to create a link between the multiple hierarchical levels of operating units. Within a sub-operating unit, one or more courses are defined in step 276. As part of defining a course, the course objectives are aligned with higher-level goals and objectives, in step 278, so that a link between courses and the meeting of certain goals and objectives can be established. In addition to the course having its own objectives, the course can also include a definition of its content (e.g., assignments, tests, reading material, collaboration exercises, etc.). A description of the various content for a course may include which course or unit objective it is aligned with. Thus, in step 280, course content is aligned with course objectives and, possible, higher-level goals and objectives. As a result, top-level managers can establish which specific resources satisfy strategic goals and objectives, while low-level managers can identify deficiencies in course offerings and content when compared to desired curriculum objectives. One advantageous benefit of the present system, described below, is that at each level of the process an appropriate form can be presented to the user to direct the definition of the assessment project.
Goals, Objectives, Standards, and Assessment Planning
-
- University
- College of Liberal Arts
- English Department
that has been originally defined for this institution or changed through subsequent modification. Such a hierarchy can, for example, be defined by the institution when initially configuring the assessment system or modified during a later update of that definition. From this “home page” a user may start a new assessment project or interact within a current assessment project.
- English Department
- College of Liberal Arts
- University
A number of sub-windows may be provided within the opening window 300. Some exemplary windows, as shown in
The “Goal Manager” window of
This list of processes may also include an identification of an appropriate term 316 to which each process relates. For example, this allows historical mission/vision statements to be maintained even as new ones are developed. The current user is also provided with a respective icon 317 that is associated with each process that indicates that user's access privileges for the particular process.
From the list 318, a user may choose to define or edit a mission/vision statement for the operating unit or program. As seen in
The text-box 322 allows the user to enter the language of the mission statement. In this instance, the vision statement is something different than the mission statement and so a separate text box 320 allows the user to enter the vision statement for the department. Once the mission and vision statements are created, the goals for the department can be defined using an interface such as the one shown in
This interface screen provides a list of the department goals that includes a label 324 for each goal and a portion of the text 326 of each goal. To add a new goal, an “Add” button 325 is provided. Although not shown, the add process would typically include defining a label for the new goal, a textual description of the goal, and some additional meta-data such as its priority, a category identifier, a keyword (to assist with searching or customized reports), and an identifier of related goal or objectives. These additional meta-data may also be included in the list via a user-controlled selection process 327. In many instances, entering the mission and vision statements for the department, as well as the goals, may already have been performed and the user may bypass such steps and move directly to defining a new assessment project.
The screen of
There are many possible customizable attributes or meta-data that can be associated with an objective within an assessment project; these attributes are defined in the various templates for an Objective from (see
The alignment of objectives with other goals and objectives is one advantageous benefit of the present system. The selection box 352 allows the user to select various available objectives and goals that have been defined. For example, departmental goals, institutional goals, and other operating unit goals and objectives can all be selected using the drop down box 352. Once the particular level of goals is selected, the specific goals can be selected using the arrow button 354. The selected goals and objectives are shown in the box 349. This process allows the user to identify other goals, objectives, or standards are aligned with the present objective being defined. Thus, an association can be created between the goals of any of the different hierarchical levels of operating units defined within the assessment project and a newly defined objective. One result is that a user at the institutional level (or some other level) can identify which lower level objectives relate to which of the strategic goals and objectives of the institution. In general, the result is that associations or alignments may be accomplished between goals and objectives of higher, lower, and parallel hierarchical levels within the organization or institution.
Within the window 349, an external standard is shown as one of the selected goals. External standards are not necessarily defined by the institution but may be defined by a regulatory body or other third party such as an accreditation council. In such a case, the drop down selection window 352 would also allow the user to select from among available external standards and have the standard's goals listed for possible selection. These standards may change over time and, therefore, can be imported into predetermined storage locations so that the interface screen of
Other areas 353, 355 of the interface screen allow the user to provide comments or attachments that are related to the objective. Such areas 353, 355 may be particularly helpful when the objective is being reviewed by a committee or other party.
Operating Units
The above-described defining of goals, assessment projects, and objectives relies on the defining of operating units. The present system and methods provide techniques for defining operating units in a flexible manner that allows an institution to model whatever its organizational hierarchy structure looks like. The interface screen of
The screen of
The identification and assignment of roles to different users is important to determing the access privileges to various data resources previously described. The screen of
The screen of
Curriculum Planning and Mapping
The previously described interface screens relate to the general category of goals and assessment planning for an operating unit. Such planning usually takes place at the relatively higher layers of the institution and relate to the strategic goals and objectives of the institution. As shown, however, lower-level operating units can define objectives and goals as well. Finally, at the lowest hierarchical levels, the objectives which are defined relate back, or are aligned to, the previously defined goals. At a higher education institution, for example, the levels can be categorized as institutional level, program level, and classroom level. For example, within a program, a number of courses or sections may defined that are designed to satisfy the objectives of the program, the department and the institution. The following figures in
Retuning quickly, to
The interface screen of
The “definition” attribute 404 allows a course objective to be defined in an efficient manner. For example, a particular objective may be defined for a course (e.g., CS101) that will also be an objective for every section of that course irrespective of the specific instructor for the section. When the section-specific objectives are defined for a section, there is no need to re-define the particular course objective but rather simply select it from all the available objectives presented on a template during the definition process. In addition, a default, or pre-selected, objective may be presented to the user as well. By selecting an already defined objective, all the attributes can be re-used and, optionally, changed by the user if desired. This functionality also allows objectives defined for one course to be selected for inclusion in an entirely different course without repeating the entire objective definition process.
An objective may be aligned with any previously defined objective or goal regardless of the operating unit in which that objective or goal was defined (for example, see
The rubric attribute 408 identifies the rubric used to measure the findings related to the objective. The present system includes functionality for building rubrics related to course objectives and for other uses within an assessment project. One of ordinary skill will recognize that a variety of techniques and methods can be used to identify the subject matter of the rubric and arrange and define its structure. Once created, these rubrics are managed and stored by the present system to be available for various uses in their respective operating units. Also, assuming proper access privileges are defined, rubrics may be shared between different operating units and assessment projects as well. The rubrics may be general in nature such as a “writing assignment” rubric that can be applied to any writing assignments. The objectives for a particular writing assignment may include, for example, “utilize research material” and “critical thinking”. The rubric used will have a result that measures attainment of these objectives.
Another level of assessment planning that can occur is at the lesson level. In other words, the lessons are identified within a course and aligned with the course or section objectives. Thus,
Within the definition of a lesson plan, there can also be the identification of the number, names, and types of assessments that are included. These assessments can include tests, quizzes, papers, collaborative assignments, on-line surveys, etc.
Once the course objectives and the lesson plans have been defined, a variety of curriculum planning and curriculum mapping investigations can be performed. Returning briefly to
More advanced curriculum planning and mapping may be performed as well using the information that the user has entered about goals, objectives, courses and lessons. For example,
Another possible curriculum mapping function can include a different view of courses and aligned objectives. For example, an operating unit that “owns” a course (e.g., the Computer Science Department) can use the information described above to identify all other units that have aligned program objectives to that course. Thus, the curriculum plan can be viewed from both a top-down approach as well as a bottom-up approach. Potential for new courses or changes to existing courses within a program can be determined by identifying which goals and objectives are being ignored by the offerings within a program. Furthermore, within the courses themselves, lesson plans can be modified or augmented to more closely align with the course objectives (which indirectly align to the other levels of the hierarchy). From the top level of the institution, evidence can be automatically gathered from across the various operating units of how the institution's goals and objectives are being met. This evidence is collected automatically, systematically, and objectively during the performance of the assessment projects and is readily available for internal evaluation by the institution as well as by external accreditation boards.
Instrument Creation and Management
In step 502, a user identifies a new test instrument through the user interface. This is typically performed by making a selection to create a new test or to edit an existing test. The user continues, in step 504, by defining the contents of the test, or instrument, itself. Because of the hierarchical arrangement of some of the operating units within the present system, it is advantageous that a test can incorporate default information required by a department or the institution. For example, every test may require the student to sign a “honor” clause or to answer certain demographic information. Thus, when a new test is defined, it will automatically include such information. In the example where the instrument is a survey, the institution may have certain required questions and each department may have its own auxiliary questions. Thus, the starting “canvas” for a survey instrument may initially have a variety of default questions depending on the unit under which the instrument is created.
In step 506, the recipient list for the instrument is selected. This list may be generated based on data about user, units or courses. The list might be generated dynamically at the time of deployment or have been generated previously and saved as a static list. For example, the recipient list may be defined as all 400-level sections taught by a tenured professor. If generated dynamically, that list would show all 400-level sections taught by a tenured professor at the time of deployment, therefore the dynamically generated list might be different when used the Fall of 2006 and the Fall of 2007, as the sections taught by tenured faculty will have changed. However, if generated statically in the Fall of 2006, it will store all 400-level sections taught by a tenured professor at the time of the list creation. That list would be the same in Fall of 2007 as it was in Fall of 2006, because it represents a static view of the data as it was at the time of list-generation. Furthermore, an instrument may be copied and then modified such that the modified copy is deployed to an earlier population or a different population altogether. Accordingly, both the original instrument and the changed copy are available for deployment in the future.
Deployment of an instrument can occur in a variety of ways, some of which rely on the academic system 116 of
Adding a new test instrument is initiated by selecting the “Add” button 519. Doing so causes a new test creation page to be presented to the user for modification and editing. From this from a user can define such items as the questions of the test, how it is scored, the instructions for completing the test, and similar test characteristics.
The present assessment system 102 also provides, via its user interface, methods for managing course evaluation instruments for a particular operating unit. For example, the interface screen 540 of
The user can elect to create a new course evaluation by selecting the “Add” button 552. Similar to techniques already described, an appropriate authoring form is provided to the user in order to define the new evaluation. An exemplary authoring form is depicted in the interface screen 560 of
Because the assessment system 102 already includes a variety of information and data that is typically used to perform questionnaire functions within the institution, a questionnaire manager may be provided that defines standard questionnaires and their contents. Thus, when a user desires to generate a questionnaire, it can be initially created within the assessment system 102 using the templates maintained by the questionnaire manager. The benefits of such a system include the advantageous result that a department, for example, will receive similarly formatted reports and content from each of its separate programs. The exemplary interface screen of
One of ordinary skill will recognize that there are similarities between a survey and other instruments for collecting data. Unlike a survey, some data collection instruments are not delivered to a specific set of users and may be open for anyone to complete. For example, a satisfaction survey would be sent to a specific set of users typically all at the same time and users would only be able to submit once. However, a curriculum change request form would be available for a user to complete at any time, as determined by the user's need to request a change in the curriculum (e.g., a faculty requesting a new course be added to the curriculum), and the user may submit it multiple times as the need to provide the data may arises on multiple occasions. The data collection capabilities of the survey and course evaluation tools can be used for such general purpose data collection instruments as well, with a change to the deployment model. Instead of being deployed to a specific set of users, such a general purpose data collection instrument could be made accessible to a set of users for submission at the discretion of the user based on the user's needs. The exemplary interface screen of
The availability of data across a wide variety of units, courses, and users raises concerns of uniform consistent and efficient evaluation of the data. Thus, an evaluation portfolio manager is provided for an operating unit that lists evaluation templates available to users within the operating unit. Typically, within a higher-education institution, for example, the evaluation template manager would be at a relatively high level such as at the college or department level. An evaluation portfolio relates to a particular portion of an assessment project. For example, a college may have to satisfy three different accreditation boards to be “accredited”. A different portfolio could be created for each of the different accreditation boards. Each portfolio would have identified, therein, a series of objectives (as outlined by the accreditation board). Drilling down into each objective would be an identification of how “satisfaction” of each objective is measured (a survey, a rubric for writing samples, and a test). The next step then is to collect the artifacts that pertain to each of these measures. Thus, a portfolio includes a label, a number of objectives, one or more ways to measure instruments related to each objective, and the artifacts themselves. At least three different types of assessment capabilities are provided in conjunction with evaluation portfolios. For example, a student or faculty may perform self-assessment. Additionally, a faculty member, for example, may use an evaluation portfolio to perform assessment of an individual (e.g., a student) with respect to a course objective, one or more program objectives, an institutional goal etc. Furthermore, the evaluation portfolios may aggregate results from many entities within the institution to provide assessment of institutional or program-level objectives such as, for example, accreditation criteria.
The interface screen of
Selecting the “Add” button 571 of the portfolio manager interface will reveal the interface screen of
This evaluation portfolio is deployed by a user to selected recipients using the “deploy” button 572 shown in
When defining the evaluation portfolio, the user may also be given the opportunity to define the artifacts that should be collected for each of the areas. Thus, by selecting one of the areas 580, the user may be provided with an artifact definition template that specifies attributes of the artifact (e.g., the scoring system, the fields, and the date). One example area in a portfolio may be to show that a student nurse can perform an intramuscular delivery of medicine. The artifact for documenting this area may require the name of the managing clinical nurse, a statement that the student was observed performing the injection, the name of the hospital, and the date of the observation. Another area in a portfolio may relate to satisfying a creative writing requirement. The artifact to apply to this area may include a paper that was scored according to a particular rubric. Thus, the artifacts themselves may vary greatly depending on the area 580 within an evaluation portfolio. Accordingly, the present assessment system 102 allows a user to define templates for the different types of artifacts and then, subsequently, when creating an evaluation portfolio, allows the selection of these templates to identify an artifact when describing an area within the evaluation portfolio, such as that in
A concept that is somewhat similar to a portfolio is that of an assignment binder. An assignment binder typically pertains to a particular accreditation program and collects artifacts related to that purpose over the course of one or more terms. The contents of the assignment binder can be used to support a review of a particular program or curriculum. However, unlike a portfolio, an assignment binder simply collects the artifacts without necessarily assigning them to particular areas or measuring them against predetermined rubrics.
The definition of goals and objectives, the creation of rubrics, and the collection of artifacts as described above allow automatic evaluation of a student's performance against program or course objectives. The assessment system 102 and its integration with the other campus systems 114 and the campus academic system 116 simplify sharing of information among the various systems. As a result, the integration of these systems results in the storage of rubrics, artifacts, various objectives, different goals, and a definition of the relationship or association between these different data entities. Using techniques known to one of ordinary skill, this integrated information can be organized and filtered to identify and report various aspects related to the stored data that may be pertinent to assessment within an institution.
For example, a “score-sheet” such as that depicted in
In response to the deployment of instruments and other activities, results are received from the students, in step 624, by the assessment system. The receipt of results may be through automatic submission by the students through a campus academic system 116 as described earlier or through an instructor who receives the results directly from the students. The rubrics may then be applied to the results, in step 626. Once the rubric is applied, scores can be assigned to the results for each of the defined goals and objectives so that a report can be generated (see
Reporting
The following figures depict a variety of different reports that may be generated using the assessment tools described herein and the data that is collected as a result of using these assessment tools. One of ordinary skill will recognize that the specific format and contents of each report may vary greatly without departing from the scope of the present invention. However, these reports show the great breadth and depth of the types of data collection, some of which may be automatic, and analysis provided by the assessment techniques and methods described herein. One advantageous benefit of the present system and techniques evidenced within the exemplary reports is the sharing of data from among different level operating units. For example, the data from a number of different course may be “rolled-up” into a report that is generated at a department level and this report may include data that is collected at the course or section level. Thus, a report related to a department goal or objective may include results from artifacts that were submitted at the course level even though there are intervening operating units between the two levels within the model of the institution. This sharing of data between different operating units and different levels of operating units occurs in the other direction as well. For example, when defining and identifying artifacts, instruments, surveys, portfolios, curriculum content, etc. at a lower-level operating unit attributes from higher-level operating units may be automatically inherited without the need to be explicitly defined
One example report is shown in
In addition to tabular reports, more graphical report templates are provided as well.
Using this template, a user can define the metric 722 that is going to be applied and displayed along with the class of individuals 724 to whom it applies. A description 726 is useful for identifying the metric and a selection window 728 allows the user to define the type of data that results from applying the metric. For example, the data may be “dollars”, “a percentage”, “raw numbers”, “an average”, etc. Near the bottom of the template of
It is useful, once a report has been defined and run, to be able to save the report for future reference. As shown in
For purposes of auditing and other monitoring functions, the assessment system 102 can track activities of users within the system in order to know what documents they have accessed and modified. Thus, the table of
Other Tools
There are a number of additional administrative type functions that simplify the management of various assessment projects and initiatives. These functions are not required by the assessment system 102 described herein but facilitate its operation and provide advantages to automating the processes described herein.
For example, users can be grouped into committees in order to simplify assigning roles and access privileges. Templates and other documents, when created, may have access privileges assigned based on which committee may access them, approve them, modify them, etc. By associating staff and faculty into different defined committees, the access privileges remain consistent even though faculty and staff can change. Additionally, upon creation, a committee can be provided with its own discussion board area, its own shared calendar, its own e-mail contact list, as well as its own storage repository for the documents it needs and generates. The interface screen of
Because of the automatic and systematic collections of data described herein, there will be a large number of files generated and stored. Organization of these files can be dependent on operating units, assessment projects, programs, courses, instrument type, etc.
Objectives: The objectives for the assessment project
Measurements: The measurements that will be used to measure the objective (e.g., test, survey, portfolio)
Action Plan: The ordered steps to be taken within the Assessment Project
Findings: The findings resulting from the measurements
Follow-up: The steps for a follow-up plan, if any.
The user is presented with a way to define these forms to handle each data collection process. Thus, when a user initiates the start of a new assessment project, the templates that are provided to the user are those templates and forms that are defined in this process. One of ordinary skill will recognize that other or additional forms may be used as well to define an assessment project template without departing from the scope of the present invention.
Within an institution, there may be many different operating units, some of which have different terms. For example, the medical school, the law school, and the engineering department may all have different term lengths and days. Thus, instruments or portfolios that must be completed by the end of a term or the middle of a term may have different due dates for different operating units. A term manager is therefore provided that allows a user to define different terms. In the interface screen of
One advantageous function described previously related to user lists and their use when deploying instruments and portfolios. In particular, a user list is a multi-faceted concept that refers to both its definition and its members. Thus, a user list may be defined as “all male juniors in the College of Engineering” and the members which satisfy that definition are also referred to as the “user list”. Referring back to
Once the user list is defined, the user has the option of making it a static list or a dynamic list. If it is a static list, then the user can apply the definition at a particular date to select the members of the user list. When some other process references this user list, then only those members within the original list will apply (e.g., only those members will receive a particular course evaluation). If it is a dynamic list, then the selected members are determined each time the user list is referenced. In other words, a survey deployed to a particular dynamic user list may be sent to different members if sent on two different dates.
The user manager interface screen of
Similar to a user list, the assessment system 102 also provides the user with a way to selectively define course lists. A course list includes those courses that match a specified set of criteria. The attributes of a course (to which the criteria are applied) may be specified and stored within the assessment system 102 or be imported from another campus computer system such as an online catalog or similar database. The breadth of different course lists that can be created is limited only by the possible number of permutations allowed by the different attributes used to describe a course. Thus, very finely defined course lists may be created to allow selective deployment of portfolios, course evaluations, and other surveys. The course list manager interface screen of
Because operating units may be hierarchically arranged, it is possible to take advantage of that organization to simplify the deployment of certain instruments/survey and to perform certain reporting. For example, in an institution made of multiple campuses, each with multiple colleges, it is possible to ensure the same survey is deployed to all colleges of a campus simply by specifying the campuses without the need to specify each individual college. This functionality is provided by allowing the user to define unit lists. A unit list is simply those operating units that meet particular criteria. The interface screen of
By automating the process of defining and collecting information related to assessment projects, a vast data warehouse can be generated that provides many different views of the assessment process. Searching through that data warehouse using selectable criteria allows a user to create a view of the data that may be useful for a variety of different purposes. One of ordinary skill will recognize that there are numerous ways to categorize and store data to aid in searching and locating pertinent information. The search interface screen of
Assessment System and Academic System
The previously-described functionality of the assessment system 102 has been focused on the assessment system in isolation from many of the other systems available within the institution or organization. However, as briefly mentioned with respect to
In
Specific to the assessment system functionality, two modules 910 and 912 are displayed on the user's interface page 900. The module 910 describes an assessment exercise that a student must complete by a certain date. It includes a link 911 that the user can select to be taken to the assessment exercise. The implementation of the exercise may be on some other computer system that receives and records the user's responses. These responses and scores are automatically provided to the assessment system 102 and associated with that particular user for that particular assessment exercise. The other module 912 relates to one or more surveys that the user is to complete. Similar to the assessment exercise, the student selects and then completes the survey with the results automatically returned to the assessment system.
The modules 910, 912 refer to instruments that are defined by a user of the assessment system 102 using the techniques previously described. The user of the assessment system 102 also employs the user list or course list (or both) to select the group of students to receive the respective instruments and then deploys the instruments. Once deployed, the assessment system 102 sends notification to the academic system to locate those students selected to receive the instrument. Within the academic system, a notation is made within the records associated with the implicated students regarding the newly available instrument. When the students next log in to the system, their respective home pages are rendered and the included notation results in a display of the appropriate modules 910, 912. An alternative deployment method would be to add a new instrument within the course area of a particular course. Thus when a student drills down to that course, the course page will be rendered and it will display a link to any new instruments.
Faculty members also use the academic system 116 and may be interested in assessment projects within their department or involving the student population that they interact with. For example, window 920 of
As mentioned, within the course-specific areas that a student may access from their virtual space within the academic system, a student has access to a variety of information pertaining to that course such as assignments, reading lists, discussion boards, lecture notes, etc. With the connection to the assessment system 102 in place, the student may now also be provided with a listing of the goals and objectives aligned to each course. Additionally, by drilling down into each lesson within the course, the student can see what are the objectives of today's or some other day's lesson. Additionally, objectives and goals can be aligned to individual content items within the course. For example, a particular discussion board topic may be presented to satisfy a particular course or department objective. This alignment can be specified by the instructor and displayed within the course area visible to the student.
The academic system 116 includes both a content management system and an administrative system. The assessment system 102 described previously can advantageously utilize this functionality already provided by the academic system. For example, the assessment-related content may be organized by operating unit and managed and accessed using the shared content window of the academic system.
Template Driven
In the above-described assessment system 102, the use of templates has been highlighted as an advantageous technique for defining and collecting the data to be used in assessment projects. For example, at the institutional level a template is provided which guides the user into providing strategic goals and mission and vision statements. When an assessment project is begun, the user is presented with templates that allow objectives to be identified, measurements to be defined, and action plans to be specified. When the identified objectives are to be defined, a template is provided to guide the user through this process and to align them with higher level goals and objectives. At the unit level, templates are provided to define portfolios, instruments, programs and program objectives. While at the course level, other appropriate templates are presented to a user to help develop lesson plans, instruments and course and content objectives. In each instance, the lower level objectives and outcomes can be aligned with higher-level objectives and goals. As a result, a template driven system is provided that defines objectives at least at three different hierarchical levels and allows automatic alignment of the objectives throughout the three levels. Of course, more levels are possible as well, because the present assessment system provides operating units with the freedom to model their organization an a flexible manner.
Workflow
The different phases of an assessment project can be considered as events within the assessment system 102. As such, a workflow system may be included to monitor and facilitate an assessment project. For example, certain re-occurring assessment projects may be tied to the start or end of a term. Thus, a calendar function within a workflow process can initiate a creation of an appropriate assessment project template and forwarding it to the initial user responsible for managing the project. For such a project, as well as those manually initiated, the subsequent steps of the assessment project may be handled by a workflow process. The task plan described earlier can include with each task the identity of the responsible parties and the disposition of the project after completion of each task. For example, once an exemplary assessment project reaches the stage where all of a department's objectives have been defined, then a subsequent step, for example, may be to create a curriculum map for that department. Through use of a workflow process this automatic stepping through the task plan may be accomplished. Some workflow steps may merely include sending e-mails to committee members regarding the status of some phase of an assessment project. Another benefit of implementing a workflow system is that parts of the assessment project may have its access privileges automatically modified at each step of the task plan. For example, once a set of program objectives are agreed upon by the committee, then these objectives become read-only and no one has permission to modify them Thus, because of the automatic and systematic collection of assessment-related information and tasks available within the assessment system 102 described herein, an automatic workflow capability may be included to further automate and monitor an institution's assessment initiatives.
Export and Import
Assessment projects, and portions thereof, are data entities in and of themselves. For example, an English Department can create an assessment project that relates to how its courses align with its objectives. The creation of such a project includes a definition of objectives and goals, an organization of operating units, a definition of courses, and an identification of course objectives. Inherent in this data is how all the objectives from the different hierarchical levels align with one another. If further detail is desired, each course can include information on course contents and instruments deployed within each course. Another assessment project may be related to how a department prepared for a particular accreditation review. This might include evaluation portfolios, surveys, and other components. In each of these cases, other institutions may benefit from copying those assessment plans. Thus, portions of an assessment project (or the entire project) may be copied within the institution for use by various programs and departments while such functionality may also occur across different institutions as well.
Thus, the presently described assessment system 102 can export an assessment project or only portions of an assessment project as a data entity that can be transferred to another institution having its own assessment system. One example of a portion of an assessment project that may be exported, or imported, may be a series of templates related to National Council for Accreditation of Teacher Education (NCATE) accreditation. At that second institution, then, the assessment system can import the data entity and have a pre-populated assessment plan to use or to modify. At a smaller level, a publisher of a course or program of courses can define the objectives met by its courses and their respective content as well as the instruments and, possibly, portfolios that might accompany such a program of courses. This information may be imported into the assessment system at a department or program level at the institution to avoid developing the same material on their own from scratch.
Within such imported or exported data there may be references to third-party standards. If a user imports a project that includes a reference to, for example, an objective of the standard (e.g., analytical reasoning), that user will most likely desire to know exactly what that standard says about that objective. Thus, importing and exporting data within the assessment system 102 provides the additional functionality to include external standards. This functionality may also extend to exporting and importing instruments related to measuring or evaluating these standards as well.
One result of having such import and export functionality included within the present assessment system is that repositories of assessment project-related data entities may be created. Thus, assessment projects, templates, instruments and other data may be exported to and stored in a repository that is accessible by other users via a network or other computer system. Such a repository of information may include different levels of anonymity. For example, an institution may want to export to the repository information about the incoming freshman class' SAT scores, high-school GPA, and student age that was collected during a particular assessment project. This information can then be used by other institutions for comparison and analysis. In such an example, the first institution may want to have their identity associated with the data but also insure that the individual student information is anonymous. In a second example, the institution itself may want to remain anonymous in interacting with the repository. In this case, the institution may provide one or more aspects of an assessment project to the repository which can then provide a comparison against a benchmark or other standard. In this way, the repository provides the institution with feedback regarding the assessment project (e.g., whether or not the students performance on a battery of tests were below, at, or above a nationwide average). Alternatively, the institution can access the repository and import different benchmarks and other standards to perform their own comparison and analysis. Because virtually any aspect of an assessment project may be exported to the repository, institutions can exchange and compare various instruments, rubrics, surveys, templates, reports, as well as results collected in performing the assessment project. Furthermore, this repository may also include non-assessment related information and data as well such as, for example, course content and individual academic portfolios.
Multi-Institution
Not only can the assessment plans and portions of the assessment plans (e.g., the objectives, the courses) be exported and shared, the results and findings can be shared as well. Accordingly, a multi-college consortium, for example, can request and receive findings from its members and then report on the findings from the overall perspective. If the members of the consortium utilize substantially similar assessment projects and initiatives, then the collected data and findings will be consistent in nature and format and will allow them to be simply amalgamated and analyzed.
A number of variations to the specific behaviors and steps described in the above examples may be made without departing from the scope of the present invention. The various illustrative logical blocks, modules, circuits, elements, and/or components described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic component, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine.
The methods or algorithms described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor.
The previous description is provided to enable any person skilled in the art to practice the various embodiments described herein. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments. Thus, the claims are not intended to be limited to the embodiments shown herein, but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” All structural and functional equivalents to the elements of the various embodiments described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. No claim element is to be construed under the provisions of 35 U.S.C. §112, sixth paragraph, unless the element is expressly recited using the phrase “means for” or, in the case of a method claim, the element is recited using the phrase “step for.”
Claims
1. A method for selectively assembling an assignment binder including a selected subset of a plurality of artifacts, the method comprising the steps of:
- receiving first input defining a criteria, the criteria related to a plurality of identifying attributes associated, respectively, with each of the plurality of artifacts;
- identifying the selected subset based on those of the plurality of artifacts having associated identifying attributes matching the criteria, wherein the criteria relates to at least two criterion related to one or more of demographic attributes, course-related attributes, operating unit-related attributes, section-related attributes, goal-related attributes, and educational experience-related attributes; and
- collecting the selected subset of artifacts within the assignment binder.
2. The method of claim 1, wherein the criteria relates to demographic attributes about an owner of each of the plurality of artifacts.
3. The method of claim 1, wherein the criteria relates to course-related attributes associated with courses associated with the plurality of artifacts.
4. The method of claim 1, wherein the criteria relates to operating unit-related attributes associated with one or more units with which the plurality of artifacts may be associated.
5. The method of claim 4, wherein an operating unit comprises one of an institution, an educational experience, a section, a college, a department, a program, and a course.
6. The method of claim 1, wherein the criteria relates to one or more respective goal-related attributes related to each of the plurality of artifacts.
7. The method of claim 6, wherein the one or more goal-related attributes include at least one of a course goal, a course objective, a program goal, a program objective, a department goal, a department objective, an operating unit goal, an operating unit objective, an external standard; and a third-party accreditation requirement.
8. The method of claim 1, further comprising the step of:
- sampling the selected subset substantially randomly as part of a determination of which artifacts in the selected subset to collect.
9. The method of claim 1, wherein each of the plurality of artifacts is a response to a respective instrument deployed to a plurality of users.
10. The method of claim 9, further comprising the steps of:
- forwarding the respective instruments to a separate computer-based system configured to disseminate information to the plurality of users; and
- receiving the plurality of artifacts from the separate computer-based system.
11. The method of claim 10, wherein the separate computer-based system comprises a campus academic system.
12. The method of claim 11, wherein the campus academic system is configured to provide an environment to disseminate course-related information to a plurality of students of an educational institution.
13. The method of claim 1, further comprising the step of:
- collecting evaluation data associated with each artifact within the selected subset.
14. The method of claim 1, further comprising the step of:
- storing the criteria as a named search, wherein another assignment binder can be subsequently assembled using the named search without the criteria being input again.
15. A method for selectively assembling an assignment binder, for a multi-level institution, including a selected subset of a plurality of artifacts, the method comprising the steps of:
- receiving first input defining a criteria, the criteria related to a plurality of identifying attributes associated, respectively, with each of the plurality of artifacts;
- forwarding the criteria to a separate computer-based system of the institution, wherein the separate computer-based system is configured to store the plurality of artifacts; and
- receiving from the separate computer-based system the selected subset of those of the plurality of artifacts having associated identifying attributes matching the criteria.
16. The method of claim 15, further comprising the step of:
- collecting the selected subset of artifacts within the assignment binder.
17. The method of claim 16, wherein the separate computer-based system comprises a campus academic system.
18. The method of claim 17, wherein the campus academic system is configured to provide an environment to disseminate course-related information to a plurality of students of an educational institution.
19. The method of claim 15, further comprising the step of:
- storing the criteria as a named search, wherein another assignment binder can be subsequently assembled using the named search without the criteria being input again.
20. The method of claim 15, further comprising the step of:
- collecting a substantially random sample of the selected subset within the assignment binder.
21. A system for selectively assembling an assignment binder including a selected subset of a plurality of artifacts, comprising:
- a programmable computer configured to:
- receive first input defining a criteria, the criteria related to a plurality of identifying attributes associated, respectively, with each of the plurality of artifacts;
- identify the selected subset based on those of the plurality of artifacts having associated identifying attributes matching the criteria, wherein the criteria relates to at least two criterion related to one or more of demographic attributes, course-related attributes, operating unit-related attributes, section-related attributes, goal-related attributes, and educational experience-related attributes; and
- collect the selected subset of artifacts within the assignment binder.
22. The system of claim 21, wherein the criteria relates to demographic attributes about an owner of each of the plurality of artifacts.
23. The system of claim 21, wherein the criteria relates to course-related attributes associated with courses associated with the plurality of artifacts.
24. The system of claim 21, wherein the criteria relates to operating unit-related attributes associated with one or more units with which the plurality of artifacts may be associated.
25. The system of claim 24, wherein an operating unit comprises one of an institution, an educational experience, a section, a college, a department, a program, and a course.
26. The system of claim 21, wherein the criteria relates to one or more respective goal-related attributes related to each of the plurality of artifacts.
27. The system of claim 26, wherein the one or more goal-related attributes include at least one of a course goal, a course objective, a program goal, a program objective, a department goal, a department objective, an operating unit goal, an operating unit objective, an external standard; and a third-party accreditation requirement.
28. The system of claim 21, wherein the programmable computer is further configured to:
- sample the selected subset substantially randomly as part of a determination of which artifacts in the selected subset to collect.
29. The system of claim 21, wherein each of the plurality of artifacts is a response to a respective instrument deployed to a plurality of users.
30. The system of claim 29, the programmable computer further configured to:
- forward the respective instruments to a separate computer-based system configured to disseminate information to the plurality of users; and
- receive the plurality of artifacts from the separate computer-based system.
31. The system of claim 30, wherein the separate computer-based system comprises a campus academic system.
32. The system of claim 31, wherein the campus academic system is configured to provide an environment to disseminate course-related information to a plurality of students of an educational institution.
33. The system of claim 21, wherein the programmable computer is further configured to:
- collect evaluation data associated with each artifact within the selected subset.
34. The system of claim 21, wherein the programmable computer is further configured to:
- store the criteria as a named search, wherein another assignment binder can be subsequently assembled using the named search without the criteria being input again.
35. A system for selectively assembling an assignment binder, for a multi-level institution, including a selected subset of a plurality of artifacts, comprising:
- a programmable computer configured to:
- receive first input defining a criteria, the criteria related to a plurality of identifying attributes associated, respectively, with each of the plurality of artifacts;
- forward the criteria to a separate computer-based system of the institution, wherein the separate computer-based system is configured to store the plurality of artifacts; and
- receive from the separate computer-based system the selected subset of those of the plurality of artifacts having associated identifying attributes matching the criteria.
36. The system of claim 35, wherein the programmable computer is further configured to:
- collect the selected subset of artifacts within the assignment binder.
37. The system of claim 36, wherein the separate computer-based system comprises a campus academic system.
38. The system of claim 37, wherein the campus academic system is configured to provide an environment to disseminate course-related information to a plurality of students of an educational institution.
39. The system claim 35, wherein the programmable computer is further configured to:
- store the criteria as a named search, wherein another assignment binder can be subsequently assembled using the named search without the criteria being input again.
40. The system of claim 35, wherein the programmable computer is further configured to:
- collect a substantially random sample of the selected subset within the assignment binder.
41. Computer readable media containing programming instructions for selectively assembling an assignment binder including a selected subset of a plurality of artifacts, that upon execution thereof, causes one or more processors to perform the steps of:
- receiving first input defining a criteria, the criteria related to a plurality of identifying attributes associated, respectively, with each of the plurality of artifacts;
- identifying the selected subset based on those of the plurality of artifacts having associated identifying attributes matching the criteria, wherein the criteria relates to at least two criterion related to one or more of demographic attributes, course-related attributes, operating unit-related attributes, section-related attributes, goal-related attributes, and educational experience-related attributes; and
- collecting the selected subset of artifacts within the assignment binder.
42. Computer readable media containing programming instructions for selectively assembling an assignment binder including a selected subset of a plurality of artifacts, that upon execution thereof, causes one or more processors to perform the steps of:
- receiving first input defining a criteria, the criteria related to a plurality of identifying attributes associated, respectively, with each of the plurality of artifacts;
- forwarding the criteria to a separate computer-based system of the institution, wherein the separate computer-based system is configured to store the plurality of artifacts; and
- receiving from the separate computer-based system the selected subset of those of the plurality of artifacts having associated identifying attributes matching the criteria.
Type: Application
Filed: Apr 12, 2006
Publication Date: Oct 26, 2006
Inventors: David Yaskin (Arlington, VA), Greg Ritter (Washington, DC)
Application Number: 11/405,030
International Classification: G06F 9/44 (20060101); G06F 17/50 (20060101);
