Automated generation of software applications using analysis of submitted content items
Based on an analysis of example content items, a proposed initial version of an application to be generated on behalf of a client is determined. At least a portion of a software program implementing the application is generated, and an indication of the software program is provided to the client. Based on input received from the client, additional operations associated with the program are initiated.
Latest Amazon Patents:
In recent years, more and more software applications are being run using smart phones and other small-footprint devices, or in cloud-based data centers, thereby reducing the need for application owners to maintain expensive application execution environments at their own premises. As application users get accustomed to easy-to-use applications, the pressure on business organizations and governmental entities to make their products and services accessible via such applications increase. Many non-business activities, such as social gatherings and the like, are also increasingly being organized and managed using phone-based or cloud-based applications.
Large well-financed organizations may be able to hire enough programming staff to develop and maintain the applications for various target environments, or outsource the work of developing and maintaining the applications. However, for a “long tail” of relatively small organizations or individuals, the costs associated with developing full-fledged applications may sometimes be prohibitive. For any given application, several different types of programming skills may be required: for example, front-end or user interface design expertise for various types of devices expected to be used for the application, back-end or database skills, security-related expertise, networking skills and so on may all be essential.
For many small organizations, making their unique service and/or product offerings accessible via easy-to-use applications may potentially lead to substantial increases in market size and/or revenue. Some small business owners may have ideas for several different types of application and channels which they would like to try out for delivering their products or services, but may lack programming skills needed to develop the applications themselves. Orchestrating the creation and deployment of such applications may present a non-trivial technical and logistical challenge.
While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include,” “including,” and “includes” mean including, but not limited to. When used in the claims, the term “or” is used as an inclusive or and not as an exclusive or. For example, the phrase “at least one of x, y, or z” means any one of x, y, and z, as well as any combination thereof.DETAILED DESCRIPTION
Various embodiments of methods and apparatus for automated generation and deployment of software applications based at least partly on the analysis of content items indicated by an application requester are described. In at least some embodiments, one or more of the techniques described herein may be implemented at a network-accessible service, which may be referred to as an automated application generation service (AAGS) or an application builder service. A customer or client of such a service, such as a businessperson who may not necessarily be skilled in at least some aspects of software development, but who does have some high-level goals about a potentially useful software application, may utilize the service to generate and/or refine the desired application without writing any software code in various embodiments. The client may begin the process of generating a particular application associated with a problem domain of interest by submitting one or more examples of content items (e.g., photographs, emails, or the like) which may (from the perspective of the client) provide hints or suggestions regarding the desired application to the service. For example, if a client is a hair salon owner or manager who wants to obtain an application associated with scheduling of hair styling appointments, a set of photos of hair styles, and/or of customers having their hair styled or cut in the salon, together with some calendar entries for hair-related appointments, may be submitted to the service. Any of a number of easy-to-use interfaces may be employed to submit the content—for example, interfaces similar to those used for sharing photographs taken on a smart phone with friends may be used, as discussed below in further detail. In at least some embodiments, the individual example content items need not necessarily be annotated in any way by the client—e.g., in the case of photographs submitted to the AAGS, the photographs may be un-enhanced or unlabeled. In other embodiments, some subset or all of the example content items submitted to an AAGS may be annotated or modified prior to submission.
Upon receiving the example content items, components of the service may analyze the content, and using any of various techniques described below such as template matching and/or machine learning, identify respective proposed initial versions of one or more candidate applications which appear to match the client's needs. A given proposed initial version may include one or more potential user interactions—e.g., a user of a hair salon scheduling application may be expected to allow clients to view available appointment slots, select a particular hair treatment and so on. An approximate visual representation of at least some such interactions may be provided by the service for one or more of the candidate applications—e.g., in the form of screenshots of a smart-phone app interface or a web-based interface which may potentially be used for the application.
At this stage, in some embodiments, customized software code for the targeted or desired application may not have been generated at the service, or only a minimal amount need to prepare the approximate visual representation may have been produced. Instead, for example, service components may utilize existing representations of user interactions of application templates or skeletons (or modify such pre-existing artifacts slightly based on the supplied example content items) in various embodiments, with the objective of obtaining an approval of one of the proposals from the client before generating a substantial amount of custom code. In the hair salon scheduling application scenario, for example, thumbnails generated from client-supplied example photographs may be used in a phase of a proposed application in which a particular hair-style is to be selected by a hair salon customer. In effect, the service may be attempting to obtain an answer from the client to a question similar to “does this sequence of interactions look (at least approximately) like the application you want?” in some embodiments. Visual representations of several options corresponding to different candidate applications may be presented to the client in some embodiments, and the client may provide feedback programmatically for the various options until eventually a set of interactions of a particular application is approved by the client. In some cases, the representations presented to the client for approval may comprise at least some non-visual elements—e.g., audio elements may be included in some application representations. (There is, of course, also the possibility that none of the choices presented by the service is acceptable to the client. In one embodiment, in such a scenario, the client may be provided access to a larger set of existing template applications and/or a source code development environment, and requested to either select one of the templates to pursue further service-performed refinements, or to provide at least an initial set of source code for the targeted application.)
After one of the proposed initial versions is approved by the client programmatically based on the presented visual representations, at least a portion of a software program corresponding to an implementation of the approved application may be generated (or retrieved from an existing library of programs) at the automated application generation service (AAGS) in various embodiments. In at least some embodiments, the AAGS or application builder service may maintain a growing repository of application templates, with which a variety of client-specific data sets may be programmatically associated for customization as needed. If the initial requirements for an application, as deduced using the content items and the client's approval of the visual representation, can be met using one of the templates, very little or no additional code may be needed at this stage of the automated application generation process in various embodiments. In other cases, the template versions of the programs may have to be modified at least slightly before being presented to the client, or a new program may have to be generated. An indication of the program (e.g., the name of the program, instructions on how to try the program out it one or more execution environments, and/or a pointer to a client-specific copy of the source code) may be provided to the client in some embodiments.
Depending on the client's goals and the client's evaluation of the initial version of the software program provided by the AAGS, a number of different follow-on operations may be initiated at the service based on requests sent by the client in various embodiments. It may be the case, for example, that the client is satisfied by the first version of the program, and wants to use it as is; in such a scenario, the client may for example either start using the application from an execution platform selected at the service, or request the deployment of the application to a different execution environment (such as a particular smart phone or server selected by the client, or a network-accessible computing service approved by the client). A variety of deployment destination environments may be available in different embodiments, as discussed below in further detail. Alternatively, in at least some embodiments the client may wish to refine the initial version of the program, e.g., by adding to the functionality, modifying the user interface, and so on, in which case the client could submit requests to the service for such changes, and an iterative process of service-generated refinements could be initiated. In some embodiments, the client may obtain access to the source code and perform at least some of the refinement-related changes (e.g., using software development expertise of the client, or of staff hired by the client). A refined version generated by the client may optionally be submitted to the AAGS for deployment at selected destinations in some embodiments. In at least one embodiment, for those clients who may desire to modify the source at any stage of the refinement process, a set of software tools such as an integrated development environment (IDE), additional debugging tools, test resources and the like may be provided by the AAGS. In some embodiments, machine learning algorithms (such as generative neural network based algorithms) may be used to refine applications—e.g., to generate at least a portion of the software for a refined version of an application. In various embodiments, the facilities and interfaces provided by the AAGS may enable clients to obtain access to high-quality applications based largely on intuitive interactions with the service, without a requirement for acquiring proficiency in software development.
Any combination of example content items of a wide variety of content categories may be supplied or indicated to the AAGS to initiate the process of generating an application, and/or during the application refinement process in different embodiments. For example, in some embodiments, photographs, videos, e-mails, contacts database entries, audio recordings, presentations, documents, calendar entries and the like may be provided. In at least some embodiments, pointers to data sources, such as uniform resource identifiers (URIs) of content on the Internet or within a storage service may be provided, e.g., in addition to or instead of the actual content examples.
When submitted content which may potentially be used to trigger the generation of an application is received at the AAGS from a client, in at least some embodiments the AAGS may transmit a confirmation request message to the client, in effect asking the equivalent of the question “do you want this content to be used to generate an application?” Additional actions at the AAGS, such as the identification of a proposed initial version of an application, may be dependent on whether the client confirms that the supplied content is to be used to generate the application. In some embodiments, clients need not submit all the content examples that are to be used at the same time—e.g., clients may submit content items in batches, informing the service programmatically when all the examples have been submitted.
In various embodiments, after the example content items are received, the AAGS may extract metadata from the content and populate at least some entries of a data set to be used for the desired application using the metadata. The data set may represent a concrete or materialized instantiation of a data model (selected by the AAGS) for the targeted application in some embodiments. For example, metadata about the dates at which various submitted photographs were taken may be used to populate a database associated with scheduling events, tags or labels associated with submitted photographs may be used to populate relationship entries in the data set, and so on. The extracted metadata may be used to select (and/or arrange in an estimated order of preference) the candidate applications whose representations are to be presented to the client for approval in some embodiments. As mentioned above, the content itself may be used to populate portions of the visual interaction representations provided to the client in some embodiments. After an initial proposal prepared at the service is approved, the data set may have to be more fully populated in order for the program corresponding to the approved application to work in some embodiments. At least in the initial stages of the interactions with the client, in some embodiments placeholder or dummy values may be generated to fill out one or more entries in the data set to enable the program to function. That is, the data set underlying the application may be populated based on a combination of information supplied by the client (e.g., in the form of metadata of the example content items, or portions of the content itself) and placeholder values selected at the service in at least some embodiments.
The AAGS may support deployment of the generated applications to a variety of destination types, and support a variety of access modes to the application, in different embodiments. For example, in some embodiments, the application may be deployed to small-footprint devices such as smart phones, to virtual or physical servers at a provider network or cloud environment, to non-provider-network servers selected by the client (e.g., servers at client-owned premises), and so on. In at least some embodiments, an application generated using the AAGS may be added to the inventory of an online marketplace or repository of downloadable applications. In some embodiments, respective versions of a given generated application may be created for several different execution environments (e.g., different server and phone operating systems) and deployed as requested by the client.
Example System Environment
The AAGs may implement a set of programmatic interfaces 177 in the depicted embodiment—e.g., a set of application programming interfaces, a web-based console, a graphical user interface and/or command line tools. In some embodiments, other programmatic interfaces such as integrated development environments (IDEs) may also be implemented for those clients who wish to modify or write source code. A subset of the programmatic interfaces may be used by clients to submit example content items 161 (such as photos, calendar entries and the like) to initiate the creation of applications in various embodiments. In at least some embodiments, the process of providing at least some categories of content items 161 to the AAGS 120 may be simplified by incorporating content submission capabilities into commonly-used interfaces of smart phones and other frequently used devices. For example, a client's smart phone camera application or photo management application may allow the phone's owner to send/share photos or videos to various destinations, and in some embodiments the AAGS may be added as one possible destination for a client 180 (after obtaining the client's “opt-in” consent), making it trivial for a client to submit selected photos without having to use a new, less familiar interface or explicitly invoke the AAGS's APIs from some other program. Similar opt-in extensions enabling easy transfers of content to the AAGS may be incorporated within file-sharing or file system applications, calendar applications, contacts applications, e-mail applications and so on for various types of devices, including for example smart phones, tablet computing devices, wearable computing devices (such as smart watches, augmented reality devices or virtual reality devices), voice-driven personal assistant devices, laptops, deskside or desktop computer systems and the like. In at least some embodiments, the example content items 161 may be submitted in an un-modified form (relative to when the items were created)—e.g. no special annotations, labels or additional guidance regarding how the items are to be used in the desired application may be required. In other embodiments, at least some of the example content items may be annotated, enhanced or modified, e.g., to help the AAGS interpret the significance or intended use of the items with respect to the application. It at least some embodiments, the set of example content items submitted to the AAGS 120 by a client may not include executable program code—instead, the content items may comprise non-executable data objects (such as image files, contacts entries, calendar entries and the like as mentioned earlier) that can be examined or analyzed using executable programs run by components of the AAGS 120.
Upon receiving a set of example application-related content items 161, a content metadata extractor component 132 of the AAGS 120 may analyze the content items. Metadata such as the types or categories of the items (e.g., whether a given item is an image, a video, a text document or the like), associated dates, times, locations and the like may be extracted from the content items in some embodiments, and provided to an initial application selector 134 in the depicted embodiment. In some embodiments, the initial application selector 134 may utilize machine learning resources 150 to identify one or more applications that appear to be potential matches for the client. For example, one or more machine learning models (e.g., deep learning models employing neural networks) may be trained in some embodiments to consume content items and/or associated metadata as input, and generate respective scores for one or more pre-existing template applications of library 102 (such as applications 112A associated with problem domain PD1, or applications 112B associated with problem domain PD2) indicating how well the template applications match inferred objectives or goals of the target application. Some subset of applications 112 with high matching scores may be selected as candidates for further interactions with the client in such embodiments. In some embodiments, machine learning models may be trained to generate the code for new candidate applications based on the input content items, e.g., instead of assigning scores to pre-existing applications. In other embodiments, the initial application selector may itself examine a pre-existing collection of template applications in library 102 and select one or more matching applications, e.g., instead of or in addition to utilizing machine learning resources 150.
Having identified or generated one or more initial candidate applications that may meet the client's target application requirements, the AAGS 120 may provide a visual representation 162 of various user interactions within the candidate application(s) to AAGS clients 180 in the depicted embodiment. The visual representations, which may for example comprise the equivalent of a set of screenshots showing what users of the proposed application may see and do as they interact with the proposed application, may be provided to obtain approval from the client for further development of the application in various embodiments. In at least some embodiments, where possible and appropriate, the client's example content items 161 (or metadata extracted from the content items) may be incorporated within the visual representations—e.g., an example photograph or e-mail address entry provided by the client may be shown in the visual representation.
If the first proposed application is not approved by the client in the depicted embodiment, representations of one or more other applications may be presented to the client, e.g., based on feedback received from the client and/or based on the matching scores generated by the AAGS. Assuming that one of the presented applications is eventually approved, and that the client 180 indicates that initial version of the application needs improvement, a process of iteratively refining the application may be begun in various embodiments. An initial set of source code may be identified or generated as the “starter” or unrefined version of the application in some embodiments, to which modifications are applied based on client feedback. Note that at least in some embodiments, source code need not necessarily be generated or stored on behalf of the client until the client approves the representation of the initial version of the application—source code-related operations may largely or wholly be deferred until after the approval in such embodiments. In other embodiments, the initial information provided to the client may include a pointer to source code for one or more applications.
An application refinement manager 136 may be responsible for interpreting the client's feedback, e.g., using machine learning resources 150 or other analytic tools, and generating modifications to the current version of the source code based on the interpretations. Multiple iterations of feedback-based refinement may be performed in some embodiments, until eventually an acceptable iteratively-refined application 163 is generated. In at least some embodiments, the client 180 may submit requests via programmatic interfaces 177 to deploy the refined application 163 to one or more deployment destinations 170. Such requests may be processed and implemented by an application deployment manager 138 in various embodiments. A number of different deployment destination types 170 may be selected by the client 180 in some embodiments, such as one or more virtual or physical servers of a provider network or cloud environment, a set of non-cloud devices specified by the client, and so on as discussed below. In addition to the deployment destinations, in some embodiments clients may also be able to specify access modes for their deployed applications—e.g., indicating the set of users (or the required roles, capabilities, permissions etc. of users) who are to be allowed to use the application. In various embodiments, during one or more stages of iterative refinement, a client may make modifications to the application source code if desired, and submit the modified source code (or an executable version of the modified source code) to the AAGS. That is, an indication of the current version (or earlier versions) of the source code (e.g., a location within a source code repository) may be provided to a client 180 by the AAGS upon request in various embodiments.
It is noted that while a network-accessible service such as an AAGS 120 may be employed in some embodiments to generate applications on behalf of clients starting with application contextual information in the form of example content items, similar techniques may be employed using a standalone tool in other embodiments. For example, in one embodiment an application builder tool capable of generating applications in a similar manner to that described above may be implemented using one or more computing devices which are not necessarily part of a network-accessible service, and not necessarily affiliated with a network-accessible service.
Example Programmatic Interactions
A set of interactions associated with generating and deploying an application may be initiated by the client submitting a group of application-related content items 220 to the service in the depicted embodiment. The AAGS 210 may transmit an application request confirmation query 222 in response to receiving the items, in effect requesting a response to a question similar to “Do you want an application using that type of content to be built automatically?” in some embodiments. Such a confirmation step may be helpful in avoiding the wastage of AAGS resources—e.g., it may sometimes be the case that content is submitted inadvertently, or an incomplete set of content examples is submitted, in which case further processing steps at the AAGS may not be required immediately. Such a conformation may also be helpful in some cases in disambiguating the interpretation of a given example content item. For example, the client could submit a photograph of a pet playing with a toy, and the AAGS may be able to determine that (a) the submission is a photograph (b) the photograph contains a pet and (c) the photograph contains a toy. As such, several object categories that could potentially correspond to operations to be performed in a targeted application may be identified in this example scenario (e.g., pets, toys, or photographs). In some embodiments the AAGS may attempt to confirm which of the categories (or combination of categories) are most relevant to the application targeted by the client.
If the client 202 responds with a confirmation 224 in the depicted embodiment, the AAGS may begin analyzing the set of example content items that was submitted. The content types of the items may be determined, and metadata transmitted along with the content items (or accessible from other sources such as web searched pertaining to the items) may be extracted in various embodiments. Based at least in part on an analysis of the content items and associated metadata, at least one proposed application which may meet the client's requirements may be identified (e.g., from a set of templates) or generated in the depicted embodiment. Other factors may also or instead be used to identify potential applications in some embodiments—e.g., information about the identity of the requesting client may be used, or other information (such as applications generated earlier on behalf of the same client, records which the client has shared with the AAGS regarding applications or devices that the client uses frequently, etc.) pertaining to the client's preferences may be used in various embodiments.
One or more visual representations 226 of typical user interactions of a rough proposed application identified by the AAGS may be transmitted to the client based on the analysis of the content and/or other factors in the depicted embodiment. The example content supplied by the client may be used in the visual representation in some embodiments to the extent possible, e.g., to indicate the manner in which the example content's purposes with respect to the targeted application have been interpreted by the AAGS. The client may eventually approve a proposed rough application, as indicated by interaction 228 in the depicted embodiment. In some cases, the approval may involve several iterations of interactions, in some of which the client indicates that a particular proposal is not acceptable (and/or why the proposal is not acceptable). In one embodiment, the AAGS may generate a list of applications ordered from high to low matching scores with respect to the client's anticipated needs, and provide respective representations of several of the applications of the list at a time, enabling the client to select among (or rank) the proposals.
After the approval 228 is obtained, an indication of an initial application implementation 230 may be provided to the client programmatically in various embodiments—e.g., a link to an executable version of the application may be provided, and/or other instructions as to how to execute or view the application may be provided. In some embodiments, the AAGS may create a client-specific initial source code version of the proposed application program—e.g., by copying/branching the source code from an existing repository of applications, or by generating at least a portion of the source code.
In response to the indication of the initial version of the application, the client 202 may provide feedback 232 (or approval 238) to the AAGS in the depicted embodiment. A number of different types of artifacts or subjective appraisals may be provided by the client as part of the feedback in different embodiments.
In one scenario, the client may find the initial version of the application acceptable as a final version, and the feedback may comprise an approval message 238 with an optional deployment request indicating the execution platforms at which the application should be deployed. In response to such a deployment directive, in various embodiments the AAGS may in some cases take one or more steps necessary to package the application for deployment, initiate the deployment to the specified destinations, and confirm that the requested deployment has succeeded by sending a deployment complete message 240 to the client.
In alternative scenarios, the client may for example try out or examine the application as presented and determine that further refinement is needed in the depicted embodiment. The client may then provide additional information which may be helpful for the refinement: e.g., details regarding the preferred look and feel of the application or the functionality of the application may be provided, additional examples of content or metadata regarding the application may be provided, and so on. In at least some embodiments, information about the configuration of notifications the targeted application should generate (e.g., under what circumstances a user should be notified by the application, what the contents of the notifications should be, and/or what the notification mechanism should be) may be provided by the client, and software code to generate the notifications may be inserted into the application by the AAGS. It is noted that in various embodiments, the feedback artifacts and assessments provided by the client may not require any programming on the part of the client—e.g., options regarding configuration of possible notification mechanisms may be provided in a drop-down menu, and the client may pick one or more of the provided options. Similarly, subjective assessments regarding the current acceptability level of various aspects of the application may be obtained using scores on a scale of 1 to 10, for example, without requiring any code to be written by the client.
As more details or subjective assessments are received at the AAGS, iteratively-refined implementations 234 of the application may be generated and indications of the improved versions may be provided to the client in the depicted embodiment. In some cases, the AAGS may submit queries for additional information to help guide the refinement changes. In various embodiments, the source code of the application may be modified at the service during the refinement iterations (some scenarios in which clients may participate in source code manipulation or creation are illustrated in
It is noted that the kinds of programmatic interactions indicated in
In some cases, clients of an AAGS (or employees/consultants of clients) may modify source code of an application generated by the AAGS.
An initial set of interactions similar to those illustrated in elements 222, 224, 226 and 228 of
The client 302 may submit a source code access request 332 corresponding to some specified version of the application (e.g., the most recent version) in some embodiments. In response, the AAGS 310 may provide source code access information 334—e.g., the credentials to access a portion of a source code repository containing the specified version of the code. In at least some embodiments, the AAGS 310 may also provide access to source code processing tools at the request of the client—e.g., one or more IDEs, debuggers, test frameworks and the like may be supported.
The client may modify the source code as desired and, in turn, provide an indication of a client-refined application version 338 in the depicted embodiment. If the client-refined application is the final version (e.g., if the client does not want any additional service-performed refinements), an optional deployment request may be transmitted as well. The AAGS may package and deploy the application as requested, and transmit a deployment completed message 340 back to the client.
In at least some embodiments, once the final version of the application is generated and approved, a client may not necessarily want the AAGS to handle deployment or any further actions. Instead, for example, the client may obtain access to the application (the source code version and/or one or more executable versions for respective types of execution platforms) if such access has not already been provided, and perform any additional follow-up actions needed. In some embodiments, at the request of a client, the AAGS may generate multiple versions of the same application compatible with, and/or optimized for, different operating environments (e.g., different operating systems, virtualization environments and so on) or different locales/geographies.
Submitted Content Categories
In at least one embodiment, the AAGS may support one or more transformation algorithms associated with the submission of example content items. For example, a filter for removing or obfuscating potentially sensitive information may be provided in some embodiments. Such a filter, if activated by a client or by default, may try to detect whether the content items being submitted may comprise financial or other private information (e.g., bank information, home addresses, mobile phone numbers, or the like) which a client may not necessarily wish to expose. If such sensitive items are detected, the client may in some embodiments be asked to confirm whether the information is acceptable without modification for further analysis by the AAGS, whether the information should be obfuscated, or whether the information should be removed from the set of information on which further processing is to be initiated. In at least some embodiments, privacy settings controlled by the clients may be used to manage sensitive information—e.g., the AAGS may enable clients to select or define the kinds of information to be considered sensitive, the actions to be taken if/when such information is detected, and so on.
Application Data Set Population
In various embodiments, an application generated with the help of an AAGS may have an associated underlying data model, and a data set representing a concrete instance of the data model may have to be generated and maintained for the application.
An instance of a data set 550 corresponding to the data model, e.g., similar to an initially-empty spreadsheet or an initially empty database, may be created in the depicted embodiment. In some embodiments, the data set may be materialized as soon as the potential target application 514 is identified, while in other embodiments the data set may not be created until the client approves an application representation presented by the AAGS. In the depicted embodiment, metadata 512 obtained from the initial example content items 510 (and/or portions of the submitted content itself) may be used to fill some of the empty slots or entries in the instance 550, as indicated by arrow 572. Other entries of the data set may initially be populated with placeholder or dummy values, as indicated by arrow 574 in various embodiments. At least some of the placeholder values may be needed for the application to work in some embodiments—that is, without a populated concrete instance of the data model 516, the application may not function properly in such embodiments. In some embodiments, the AAGS may comprise one or more placeholder value selectors 570, which are responsible for generating reasonable placeholder values which (a) enable the application to function properly and (b) are logically compatible with the entries of the data set which were populated using the supplied content items. For example, if the application deals with scheduling soccer games, and a placeholder value for the data set is to indicate a result of a soccer game to be e-mailed to some users, a placeholder result value of “2-0” or “3-1” may be more appropriate than a basketball-like score of “75-52”. In at least some embodiments, machine learning algorithms may be used to generate one or more placeholder values.
In various embodiments, during the course of further interactions with the client, as the application is being refined, some or all of the placeholder values may gradually be replaced using additional data or data sources indicated by the client (e.g., in response to queries from the AAGS). These changes may result in making the instantiated data set 550 more and more representative of the real-world usage scenarios anticipated for the application in such embodiments. In some embodiments, during the process of application refinement, the client may indicate third-party data sources (such as various types of publicly-accessible databases or encyclopedia sections accessible over the Internet) which can be used to populate various entries of the data set.
In some cases, very few example content items may have to be provided for the AAGS to deduce the type of application which is to be generated on behalf of the client. In the second example scenario shown in
Provider Network Environment
In some embodiments, software applications may be generated automatically using resources of a provider network.
In the depicted embodiment, provider network 701 may comprise resources used to implement a plurality of services, including for example a virtual computing service 703, a database or storage service 723, a machine learning service 771, and an automated application generation service 743. Components of a given service may utilize components of other services in the depicted embodiment—e.g., for some machine learning tasks, a component of the machine learning service 771 may utilize virtual machines implemented at computing platforms such as 705A-705D of the virtualized computing service, data sets for applications generated at the AAGS 743 may be stored at database/storage service 723, computing platforms of the VCS may be used as deployment destinations for applications generated at the AAGS, and so on. Input data, intermediate results, final results and/or other artifacts of various machine learning algorithms or models, some of which may be employed by the AAGS 743, may be stored at storage servers 725 (e.g., 725A-725D) of the database or storage service 723 in some embodiments. Individual ones of the services shown in
As shown, the AAGS 743 may comprise, among other components, one or more application generation coordinators 745, interaction managers 747 and/or a set of application templates 749 in the depicted embodiment. The application generation coordinators 745 may, for example, invoke algorithms selected from the machine learning algorithm library 775 to identify candidate applications corresponding to example content items submitted by clients, analyze various types of feedback provided by clients via programmatic interfaces 777 as applications are refined, and so on. The interaction managers 747 may (as indicated by their names) coordinate programmatic interactions with clients of the AAGS as well as other provider network services in the depicted embodiment. In some embodiments, as mentioned earlier, when an application based on submitted example content items is to be generated or identified by the AAGS on behalf of a client, metadata extracted from the submitted content items may be compared to metadata associated with content of various template applications 749, and one or more of the template applications may be selected as initial versions of the desired applications. In some implementations, at least a portion of a library of template applications may be stored at a database/storage service 723, and may be accessed from the database/storage service as needed at the AAGS 743. In some embodiments, the machine learning service may support both batch jobs (handled by a batch job scheduler 779 which allocates resources for the jobs and manages dependencies among jobs) and shorter real-time tasks. In at least one embodiment, the machine learning service 771 may have access to or include a set of execution platforms 776 that are optimized for machine learning tasks (e.g., platforms that have customized hardware such as GPU arrays and/or customized software stacks). Depending on the suitability of such platforms for the machine learning tasks needed for automated application generation, one or more execution platforms 776 may be employed on behalf of the AAGS 743 in the depicted embodiment.
In at least some embodiments, at least some of the techniques discussed above for generating applications without requiring source code to be supplied by clients may be accomplished using non-specialized computing platforms of the virtualized computing service 703. As mentioned earlier, the techniques for automated generation of applications may be implemented without acquiring resources of network-accessible services such as those shown in
Deployment and Access Options
In the depicted embodiment, application deployment and access options 810 may include the option of running the application on a single device 812, such as the client's own smart phone, tablet computing device or laptop, with access granted only to the owner/user of the device. This level of access may be referred to as device owner or DO access. Option 812 may be used, for example, if the client does not wish to share the application with anyone else—e.g., if the application was intended to simplify some set of tasks that are of interest only to the client. In some embodiments, the client may wish to deploy the application to a group of devices 818, such as the devices of employees who work in a particular department of an organization, or the devices of a set of friends, with DO access being granted.
In various embodiments, the application may be deployed at the client's request to a provider network service, such as the virtualized computing service shown in
In one embodiment, a set of servers at premises owned by the client (or by an organization to which the client belongs) may be selected as a deployment destination, as indicated in option 822. In one such embodiment, access control lists (ACLs) may be used to control access to the application when it is run at such servers. Other access control mechanisms may be used in other embodiments.
In one embodiment, as indicated in option 820, a version of an automatically-generated application may be included in an online application marketplace, from which it may be downloaded and run at various devices such as smart phones, tablets, and the like. The application may be downloaded without payment, or only after payment, depending on the objectives/wishes of the application owner (the client on whose behalf the application was generated). In at least one embodiment, a version of the application may be automatically deployed to a fleet 824 of Internet-of-things (IoT) devices, such as a fleet of voice-controlled assistant devices. Other deployment destination options may be supported in at least some embodiments.
Methods for Generating Software Applications Starting with Example Content Items
In various embodiments, metadata pertaining to the submission may be extracted from the example content items (element 904) and/or obtained from other sources (e.g., databases, knowledge bases or encyclopedias). The metadata may, for example, include the content type (photo, video, calendar entry etc.) as well as various characteristics or properties of the content examples (e.g., dates, locations, recognized objects types, etc.) in different embodiments. In some embodiments, the service or tool may utilize one or more programmatic interactions with the client to confirm that an application which includes operations on content categories or object types indicated by the example items is in fact to be generated.
One or more proposed initial versions of an application may be selected or generated based at least partly on the content and associated metadata (element 907). In some embodiments, the extracted metadata may be compared with respective metadata associated with a set of template applications maintained at the application generation tool/service, and the application(s) whose metadata have high degrees of similarity with the submitted content may be selected for further processing. In other embodiments, one or more machine learning models or algorithms (e.g., algorithms employing neural network-based models) may be used to generate or select the proposed application. Each candidate application may have an associated data model in the depicted embodiment, of which a concrete populated instance may be required for the application to function as intended.
An instance of a data set corresponding to the data model of a proposed initial version of the application may be initialized and populated. If possible, the provided example content items or their metadata may be used to populate at least some entries in the data set (element 910) in the depicted embodiment. Placeholder values may be used to fill out other entries (e.g., those entries for which values could not be obtained from the analysis of the submitted content) in some embodiments.
A representation (e.g., the equivalent of a sequence of screenshots) representing user interactions with the proposed initial version of the application may be transmitted to the client for approval in various embodiments. In some embodiments, the representation may include at least some non-visual elements (e.g., portions of audio may be included). Upon approval of the proposal, at least a portion of the software program code of the initial version of the application (which uses the populated data set) may be generated in some embodiments (element 916). It is noted that in some embodiments, there may be some scenarios in which no new code may have to be generated—e.g., a pre-existing application from an application builder service library may be sufficient for the client's needs; code may only have to be generated of needed.
An indication of the initial version of the program may be provided to the client in various embodiments (element 919). In some embodiments, the client may for example be permitted to try out or execute the initial version of the application (e.g., with access granted to run an executable version). In other embodiments, depending for example on the preferences of the client, access to the source code may also or instead be granted to the client.
Based on client feedback and input, additional operations associated with the application may be performed in various embodiments. For example, as needed, a sequence of one or more refined versions of the application program may be generated (element 922), based at least partly on additional content details provided by the client, notification configuration preferences indicated by the client for the target application, and so on. The process of iterative refinement based on client feedback may be continued until eventually an approved version of the application program is obtained in some embodiments. In some cases, clients may modify portions or all of the source code to implement various improvements or functional enhancements of the application as it is refined. Clients interested in modifying the software may not be required to take over modification responsibilities for the entire application source in at least some embodiments: for example, a client may make a minor change to one portion of the code, and then resume service-implemented refinement. Tools such as IDEs, source code repository management and the like may be provided by the application builder tool or service in various embodiments.
In at least some embodiments, indications of one or more deployment destination options for the automatically generated application maybe provided to the client at some stage of the interactions with the application builder tool or service. In response to a client request, an approved version of the application may be deployed to one or more selected deployment destinations in the depicted embodiment (element 925). In various embodiments, a fully functional scalable application may be generated and deployed on behalf of a client, starting with the analysis of content items submitted by the client, without requiring source code to be provided by the client.
It is noted that in various embodiments, some of the operations shown in
The techniques described above, of analyzing example content items submitted by a client, such as photographs, emails, and the like, using the results of the analysis to generate software applications on behalf of the client without requiring source code from the client, and then deploying the applications on any combination of desired application execution platforms, may be useful in a variety of scenarios. Numerous small business owners who are not experts in software development may for example obtain customized applications from an application builder service implementing the techniques. Individuals or groups who wish to organize and manage charitable events, school events, neighborhood events, small-scale sports events and the like may be able to automate part of their tasks using automatically generated applications. Even organizations or entities that possess programming expertise may benefit from the automated creation of applications, by using the application builder service or tool to implement some (e.g., less critical) parts of an application, while focusing their resources to other parts of the application where the greatest return on their in-house programming expertise can be obtained.
Illustrative Computer System
In at least some embodiments, a server that implements a portion or all of one or more of the technologies described herein, including the techniques for analyzing submitted content items, generating or selecting applications, refining applications, deploying applications and/or executing the deployed applications may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media.
In various embodiments, computing device 9000 may be a uniprocessor system including one processor 9010, or a multiprocessor system including several processors 9010 (e.g., two, four, eight, or another suitable number). Processors 9010 may be any suitable processors capable of executing instructions. For example, in various embodiments, processors 9010 may be general-purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of processors 9010 may commonly, but not necessarily, implement the same ISA. In some implementations, graphics processing units (GPUs) may be used instead of, or in addition to, conventional processors.
System memory 9020 may be configured to store instructions and data accessible by processor(s) 9010. In at least some embodiments, the system memory 9020 may comprise both volatile and non-volatile portions; in other embodiments, only volatile memory may be used. In various embodiments, the volatile portion of system memory 9020 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM or any other type of memory. For the non-volatile portion of system memory (which may comprise one or more NVDIMMs, for example), in some embodiments flash-based memory devices, including NAND-flash devices, may be used. In at least some embodiments, the non-volatile portion of the system memory may include a power source, such as a supercapacitor or other power storage device (e.g., a battery). In various embodiments, memristor based resistive random access memory (ReRAM), three-dimensional NAND technologies, Ferroelectric RAM, magnetoresistive RAM (MRAM), or any of various types of phase change memory (PCM) may be used at least for the non-volatile portion of system memory. In the illustrated embodiment, program instructions and data implementing one or more desired functions, such as those methods, techniques, and data described above, are shown stored within system memory 9020 as code 9025 and data 9026.
In one embodiment, I/O interface 9030 may be configured to coordinate I/O traffic between processor 9010, system memory 9020, and any peripheral devices in the device, including network interface 9040 or other peripheral interfaces such as various types of persistent and/or volatile storage devices. In some embodiments, I/O interface 9030 may perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., system memory 9020) into a format suitable for use by another component (e.g., processor 9010). In some embodiments, I/O interface 9030 may include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example. In some embodiments, the function of I/O interface 9030 may be split into two or more separate components, such as a north bridge and a south bridge, for example. Also, in some embodiments some or all of the functionality of I/O interface 9030, such as an interface to system memory 9020, may be incorporated directly into processor 9010.
Network interface 9040 may be configured to allow data to be exchanged between computing device 9000 and other devices 9060 attached to a network or networks 9050, such as other computer systems or devices as illustrated in
In some embodiments, system memory 9020 may be one embodiment of a computer-accessible medium configured to store program instructions and data as described above for
Various embodiments may further include receiving, sending or storing instructions and/or data implemented in accordance with the foregoing description upon a computer-accessible medium. Generally speaking, a computer-accessible medium may include storage media or memory media such as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile or non-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc., as well as transmission media or signals such as electrical, electromagnetic, or digital signals, conveyed via a communication medium such as network and/or a wireless link.
The various methods as illustrated in the Figures and described herein represent exemplary embodiments of methods. The methods may be implemented in software, hardware, or a combination thereof. The order of method may be changed, and various elements may be added, reordered, combined, omitted, modified, etc.
Various modifications and changes may be made as would be obvious to a person skilled in the art having the benefit of this disclosure. It is intended to embrace all such modifications and changes and, accordingly, the above description to be regarded in an illustrative rather than a restrictive sense.
1. A system, comprising:
- one or more computing devices of an application generation service (AGS);
- wherein the one or more computing devices include instructions that upon execution on a processor cause the one or more computing devices to: determine that an application associated with one or more example content items submitted by at least one computing device of a client to the one or more computing devices of the AGS is to be generated; generate a proposed initial version of the application, wherein the proposed initial version of the application comprises one or more proposed user interactions that are identified, by the one or more computing devices of the AGS, based at least in part on an analysis of metadata extracted from the one or more example content items submitted by the at least one computing device of the client to the one or more computing devices of the AGS; transmit, to the at least one computing device of the client, an indication of the proposed initial version of the application, wherein the indication comprises respective visual representations of the one or more proposed user interactions of the proposed initial version of the application; at least partly responsive to determining that the client has approved the proposed initial version of the application, generate a software program implementing at least a portion of the proposed initial version of the application; provide an indication of the software program to the at least one computing device of the client; and perform, based at least in part on input received from the at least one computing device of the client, one or more operations associated with a refined version of the application.
2. The system as recited in claim 1, wherein the one or more example content items comprise one or more of: (a) a photograph, (b) a video, (c) an e-mail, (d) a contacts database entry, (e) an audio, (f) a presentation, (g) a document, or (h) a calendar entry.
3. The system as recited in claim 1, wherein the instructions upon execution on the processor cause the one or more computing devices to:
- populate, based at least in part on the metadata extracted from the one or more example content items, at least some entries of a data set corresponding to the proposed initial version of the application.
4. The system as recited in claim 1, wherein the one or more operations associated with the refined version of the application comprise an operation to configure one or more notifications to be provided to users from the application.
5. The system as recited in claim 1, wherein the one or more operations associated with the refined version of the application comprise an operation to deploy the refined version of the application at a deployment destination.
6. A method, comprising:
- performing, by one or more computing devices of an application generation service (AGS): determining that an application associated with one or more example content items submitted by at least one computing device of a client to the one or more computing devices of the AGS is to be generated on behalf of the client; analyzing the one or more example content items; identifying, based at least in part on the analyzing the one or more example content items, one or more proposed user interactions for a proposed initial version of the application to be generated on behalf of the client; generating the proposed initial version of the application, wherein the proposed initial version of the application comprises the one or more proposed user interactions that are identified by the one or more computing devices of the AGS, based at least in part on an analysis of metadata extracted from the one or more example content items submitted by the at least one computing device of the client to the one or more computing devices of the AGS; transmitting, to the at least one computing device of the client, an indication of the proposed initial version of the application, wherein the indication comprises respective visual representations of the one or more proposed user interactions of the proposed initial version of the application; generating at least a portion of a program implementing the proposed initial version of the application; providing an indication of the program to the at least one computing device of the client; and initiating, based at least in part on input received from the at least one computing device of the client, one or more additional operations associated with a refined version of the application.
7. The method as recited in claim 6, wherein the one or more example content items comprise one or more of: (a) a photograph, (b) a video, (c) an e-mail, (d) a contacts database entry, (e) an audio, (f) a presentation, (g) a document, or (h) a calendar entry.
8. The method as recited in claim 6, wherein the analysis of the one or more example content items comprises:
- extracting the metadata from the one or more example content items; and
- populating, based at least in part on the metadata extracted from the one or more example content items, at least some entries of a data set corresponding to the proposed initial version of the application.
9. The method as recited in claim 8, further comprising:
- in response to determining that the metadata extracted from the one or more example content items is insufficient to populate a particular entry of the data set, determining a placeholder value to be used for the particular entry to generate the program.
10. The method as recited in claim 6, wherein the indication of the program provided to the client comprises an indication of source code of the program, wherein the input received from the client comprises a modified version of the source code, and wherein the one or more additional operations include:
- deploying an executable version of the modified version of the source code.
11. The method as recited in claim 6, further comprising performing, by the one or more computing devices:
- providing an indication of one or more deployment destination options for a particular version of the application, wherein the one or more additional operations include: deploying the particular version of the application at a selected deployment destination option of the one or more deployment destination options.
12. The method as recited in claim 11, wherein a particular deployment destination option comprises one or more of: (a) a smart phone, (b) a network-accessible service of a provider network, (c) a server indicated by the client, or (d) a collection of downloadable applications.
13. The method as recited in claim 6, further comprising performing, by the one or more computing devices:
- selecting the application from a plurality of applications, wherein said selecting is based at least in part on comparing one or more properties of a particular example content item with corresponding properties of a content item associated with the application.
14. The method as recited in claim 6, wherein said determining the proposed initial version of the application comprises:
- utilizing one or more machine learning algorithms.
15. The method as recited in claim 6, further comprising performing, by the one or more computing devices:
- receiving an indication of at least one example content item of the one or more example content items via a content submission interface of one or more of: (a) a smart phone (b) a tablet computing device or (c) a wearable computing device.
16. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors, of an application generation service (AGS), cause the one or more processors to:
- determine that an application associated with one or more example content items submitted by at least one computing device of a client to the one or more computing devices of the AGS is to be generated on behalf of the client;
- identify one or more proposed user interactions for a proposed initial version of the application to be generated on behalf of the client, wherein, to identify the one or more proposed user interactions, the one or more processors of the AGS analyze the one or more example content items;
- generate the proposed initial version of the application, wherein the proposed initial version of the application comprises the one or more proposed user interactions that are identified by the one or more processors of the AGS, based at least in part on an analysis of metadata extracted from the one or more example content items submitted by the at least one computing device of the client to the one or more computing devices of the AGS;
- transmit, to the at least one computing device of the client, an indication of the proposed initial version of the application, wherein the indication comprises respective visual representations of the one or more proposed user interactions of the proposed initial version of the application;
- obtain at least a portion of a program implementing the proposed initial version of the application;
- provide an indication of the program to the at least one computing device of the client; and
- initiate, based at least in part on input received from the at least one computing device of the client, one or more additional operations associated with a refined version of the application.
17. The non-transitory computer-accessible storage medium as recited in claim 16, wherein the one or more example content items comprise one or more of: (a) a photograph, (b) a video, (c) an e-mail, (d) a contacts database entry, (e) an audio, (f) a presentation, (g) a document, or (h) a calendar entry.
18. The non-transitory computer-accessible storage medium as recited in claim 16, wherein the instructions when executed on one or more processors cause the one or more processors to:
- obtain metadata pertaining to the one or more example content items; and
- populate, based at least in part on the metadata, at least some entries of a data set corresponding to the proposed initial version of the application.
19. The non-transitory computer-accessible storage medium as recited in claim 16, wherein the one or more additional operations include deploying a particular version of the application at a deployment destination.
20. The non-transitory computer-accessible storage medium as recited in claim 16, wherein the one or more additional operations include generating a refined version of the application using a machine learning algorithm.
|5187788||February 16, 1993||Marmelstein|
|5499371||March 12, 1996||Henninger et al.|
|6601234||July 29, 2003||Bowman-Amuah|
|6742175||May 25, 2004||Brassard|
|6877155||April 5, 2005||Lindsey|
|6901579||May 31, 2005||Suguta|
|7032210||April 18, 2006||Alloing et al.|
|7047518||May 16, 2006||Little|
|7404177||July 22, 2008||Greenfield|
|7941438||May 10, 2011||Molina-Moreno et al.|
|8938726||January 20, 2015||Barak|
|9710429||July 18, 2017||Raghunath|
|20020091990||July 11, 2002||Little|
|20020133368||September 19, 2002||Strutt|
|20030110472||June 12, 2003||Alloing|
|20040233232||November 25, 2004||Iborra|
|20060200748||September 7, 2006||Shenfield|
|20080270561||October 30, 2008||Tang|
|20080275910||November 6, 2008||Molina-Moreno|
|20090070412||March 12, 2009||D'Angelo|
|20090313245||December 17, 2009||Weyl|
|20100250497||September 30, 2010||Redlich|
|20110113424||May 12, 2011||Ewington|
|20110209133||August 25, 2011||Mahajan|
|20120265758||October 18, 2012||Han|
|20130080887||March 28, 2013||Hou|
|20170032279||February 2, 2017||Miserendino|
|20170177621||June 22, 2017||Cardonha|
|20170178002||June 22, 2017||Moriarty|
|20180014076||January 11, 2018||Shanson|
|20180014077||January 11, 2018||Hou|
|20180025161||January 25, 2018||Gauthier|
|20180068666||March 8, 2018||Riedmiller|
|20180242038||August 23, 2018||Harper|
|20190250891||August 15, 2019||Kumar|
|20200092351||March 19, 2020||Chiussi|
- Title: The Rhizomer Semantic Content Management system; author: Roberto Garcia et al; Published on 2008.
- Title: Anytime dynamic programming for coalition structure generation , author: Travis C Service et al published on 2010.
- Title: Adaptive user interface generation for web services ; author: J He et al , published on 2007.
- Lili Mou, et al., “On End-to-End Program Generation from User Intention by Deep Neural Networks,” ACM, Oct. 25, 2015, pp. 1-4.
- Tao Lei, et al., “From Natural Language Specifications to Program Input Parsers,” 2013, In Association for Computational Linguistics (ACL), pp. 1-10.
- Joel Galenson, et al., “CodeHint: Dynamic and Interactive Synthesis of Code Snippets,” ICSE'14, May 31-Jun. 7, 2014, ACM 978-1-4503-2756-5/14/05, pp. 653-663.
- Amazon web Services, “Amazon Machine Learning, Developer Guide, Version Latest,” Copyright © 2017 Amazon Web Services, Inc., pp. 1-153, downloaded Feb. 2, 2017.
- “17 Top App Makers to Create Your Own Mobile App—Without a Single Line of Code!,” May 16, 2017, downloaded from https://www.websitetooltester.com/en/blog/app-makers/, pp. 1-48.
- Mehdi Manshadi, et al., “Integrating Programming by Example and Natural Language Programming,” Copyright c 2013, Association for the Advancement of Artificial Intelligence (www.aaai.org), pp. 1-7.
Filed: Sep 25, 2017
Date of Patent: Dec 29, 2020
Assignee: Amazon Technologies, Inc. (Seattle, WA)
Inventors: Rebecca Buck (San Francisco, CA), Kenneth Chen (Palo Alto, CA), Tatyana Mamut (San Francisco, CA), Ian Scott Eslick (San Francisco, CA), Paul J. Furio (Seattle, WA), Scott Peter Perket (Oakland, CA)
Primary Examiner: Chameli Das
Application Number: 15/714,466
International Classification: G06F 9/44 (20180101); G06F 8/35 (20180101); G06F 8/60 (20180101); G06F 8/71 (20180101); G06F 8/20 (20180101);