System and method for discriminating nature of communication traffic transmitted through network based on envelope characteristics

- Reality Analytics, Inc.

A system and method are provided for discriminating the nature of traffic flowing through a computer network. Various types of traffic are distinguished. For example, traffic sourced by a software application that generates them may be distinguished. The problem of detecting malicious traffic either originating outside a target network, or inside a target network from a compromised computer is addressed, as is the problem of distinguishing malicious traffic from legitimate web surfing or other applications activity. Traffic distinctions are made based on signal envelope characteristics in such manner as to preserve robust performance even when the subject traffic is encrypted.

Skip to: Description  ·  Claims  ·  References Cited  · Patent History  ·  Patent History
Description
RELATED APPLICATION DATA

This Application is a Continuation-In-Part of co-pending patent application Ser. No. 13/665,916 filed 31 Oct. 2012, which is based on Provisional Patent Application No. 61/553,944, filed 31 Oct. 2011. This Application is also based on Provisional Patent Application No. 61/791,028, filed 15 Mar. 2013.

BACKGROUND OF THE INVENTION

The present invention is directed to a system and method for discriminating the actual origination of communication signals transmitted through a network interconnection. More specifically, the system and method are directed to the processing of communication signals received at a local, or reference, site through a network interconnection to determine and/or uniquely characterize the remote site origination of such communication signals. The system and method provide for this determination and/or unique characterization in a manner that is signal payload-agnostic, or data content-agnostic, manner. They do so by, among other things, ascertaining the envelope characteristics of the communication signals in question and classifying based thereon the remote site from which the signals originated, the legitimacy of, or other aspects of the nature of the underlying traffic.

In certain embodiments and applications, the system and method provide for such classification of remote site origin in data content-agnostic manner for communication signals transmitted from certain websites remotely accessed by a local site through the internet, namely the world wide web. In these embodiments and applications, the system and method exploit the fact that signal transmissions in certain widely employed communication protocols pass the signals in packetized data segments. Various envelope characteristics are defined by the sequence(s) of packetized data segments transmitted to the local site during particular interconnected sessions. One or more characteristic signatures may be obtained according to these envelope characteristics, so as to uniquely characterize the particular remote site originating the data segments.

There are many instances where it is desirable to know what particular website or web-service originated certain communication signals that have been received by remote access over a network, even when the address of that website or service is dynamic or is obscured for example by NAT (network address translation) or proxy forwarding. In some instances it is desirable to recognize more specifically when a particular type of session is occurring over a network—say, the use of a particular web form, transfer of data from a particular application, or the occurrence of malicious software activities over the network. Where a site in question is uncooperative with the monitoring measures in place, is deliberately evasive to such monitoring, or is particularly sensitive to privacy concerns, the site may employ encryption measures in the given communication channel to make it difficult or impractical to determine its identity based on the payload of data. Even in cases where clear-channel data is readily accessible, it may become computationally challenging to store and process necessary data signatures when there are potentially many cases of interest. Thus, there is a need for a compact, fast, and minimally invasive approach to identifying remote sites such as websites or web-services accessed through a network.

Applications of such include detection for security purposes. These include monitoring of user activities for consistency with a business or government purpose without violation of their privately encrypted data. It is desirable to detect instances when users are redirected to malicious websites, masquerading as real commercial counterparts. It is also desirable to discover cases in which malicious software on a computer communications with outside entities; in particular, where such communications may be masked as benign web traffic such as web browsing, and where such outside servers may operate behind changing apparent web addresses on order to evade detection.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a system and method for automatically and accurately distinguishing the sources from which communication signals passed through a network interconnection originate.

It is another object of the present invention to provide a system and method for automatically and accurately discriminating sources of communication signals passed through a network interconnection in data content-agnostic manner.

It is yet another object of the present invention to provide a system and method for automatically and accurately identifying and classifying the remote sites which actually originate communication signals passed through a network interconnection to a local, or reference, site based on envelope information ascertained from the communication signals.

These and other objects are attained by a system formed in accordance with certain embodiments of the present invention for distinguishing between a plurality of remote sites accessed through a network interconnection by a reference site based upon envelope characteristics of communication signals transmitted therebetween. The system comprises a capture unit time-capturing a plurality of segments of the communications signals transmitted during an interconnected session established between one of the remote sites and the reference site. A parsing unit is coupled to the capture unit to selectively generate for each of the interconnected remote sites an envelope signal indicative of at least one resource allocation response thereof during the interconnection session. The envelope signal is defined by values of at least a first predetermined envelope parameter acquired from the time-captured segments. A signature unit is coupled to receive the envelope signal from the parsing unit for each interconnected remote site to be identified, the signature unit establishing responsive to the envelope signal a characteristic signature for uniquely identifying the interconnected remote site. Newly-captured communication signal segments may be classified thereby in their remote site origination based on the characteristic signatures of identified remote sites.

A method formed in accordance with certain embodiments of the present invention provides for distinguishing between a plurality of remote sites accessed through a network interconnection by a reference site based upon envelope characteristics of communication signals transmitted therebetween. The method comprises time-capturing a plurality of segments of the communications signals transmitted during an interconnected session established between one of the remote sites and the reference site. The method also comprises parsing the time-captured segments to selectively generate for each the interconnected remote sites an envelope signal indicative of at least one resource allocation response thereof during the interconnection session, the envelope signal being defined by values of at least a first predetermined envelope parameter acquired from the time-captured segments. A characteristic signature is established for each interconnected remote site to be identified responsive to the envelope signal generated, which characteristic signature uniquely identifies the interconnected remote site. Newly-captured communication signal segments are thereby classified in remote site origination based on the characteristic signatures of identified remote sites.

A system formed in accordance with certain other embodiments of the present invention provides for discriminating remote site origination of communication signals received by a reference site from a site remotely accessed through a network interconnection, based upon envelope characteristics of the transmitted communication signals. The system comprises a capture unit time-capturing a plurality of segments of the communications signals transmitted during an interconnected session established between one of the remote sites and the reference site. The system also comprises a parsing unit coupled to the capture unit to selectively generate for each of the interconnected remote sites an envelope signal indicative of at least one resource allocation response thereof during the interconnection session. The envelope signal is defined by values of at least a first predetermined envelope parameter acquired from the time-captured segments; and, for each remote site having a resource allocation response accessing multiple resources, the parsing unit generates a plurality of envelope sub-signals each corresponding to one accessed resource. The envelope signal thereby includes a concatenation of envelope sub-signals one with the other. The system further comprises a signature unit coupled to receive the envelope signal from the parsing unit for each interconnected remote site to be identified. The signature unit establishes responsive to the envelope signal a characteristic signature for uniquely identifying the interconnected remote site, such that newly-captured communication signal segments may be thereby classified in remote site origination based on the characteristic signatures of identified remote sites. A classifier unit is coupled to receive from the parsing unit the envelope signal generated thereby for newly-captured communication signal segments from an interconnection session established between an unidentified remote site and the reference site. The classifier unit classifies the newly-captured communication signal segments in remote site origination responsive to comparison of the envelope signal thereof with the characteristic signatures of identified remote sites.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an illustrative example of data segments time-captured from packets of communication signal transmissions between local/reference and remote sites through a network interconnection in accordance with an exemplary embodiment of the present invention, tabulated in sequence;

FIG. 1B is a schematic diagram illustrating the timing of packet transfers during one example of an interconnect session between a browser and server through a network;

FIG. 1C is a longer, slightly modified version of the tabulated listing of time-captured data segments shown in the example of FIG. 1A;

FIG. 2A is a schematic diagram illustrating the flow of processes for training and classification modes of operation in a system formed in accordance with an exemplary embodiment of the present invention;

FIG. 2B is a schematic diagram illustrating the vector form of envelope sub-signals obtained from sequences of data segments captured as illustrated in FIG. 1A in the exemplary embodiment of FIG. 2A;

FIG. 2C is a flow chart illustrating the flow of processes in a method implemented by the embodiment of FIG. 2A;

FIG. 2D is a flow chart illustrating the flow of processes in a method implemented as shown in FIG. 2C, but with generalized incorporation of unspecified envelope parameters;

FIG. 3 is a set of comparative graphic plots illustrating examples of envelope sub-signals and their combined envelope signals obtained in the exemplary embodiment of FIG. 2A for communication signals transmitted between the same remote and local sites over different interconnect sessions;

FIG. 4A is a schematic diagram illustrating the flow of processes in training and classification portions of the exemplary embodiment of FIG. 2A;

FIG. 4B is a flow chart illustrating the flow of processes in training and classification portions a method implemented by the embodiment of FIG. 2A;

FIG. 5A is an illustrative graphic plot for examples of comparison values obtained during a training mode of operation by the exemplary embodiment of FIG. 2A;

FIG. 5B is the graphic plot of FIG. 5A, annotated with dividing line and threshold indicia for separating classes of graphically plotted points;

FIG. 6A is an illustrative graphic plot similar to that shown in FIG. 5A, of comparison values obtained during a training mode of operation by the exemplary embodiment of FIG. 2A, with a larger set of non-matching remote sites;

FIG. 6B is an enlarged view of a portion of the graphic plot shown in FIG. 6A, annotated with dividing line and threshold indicia for separating classes of graphically plotted points;

FIG. 6C illustrates a linear partition SVM generated in accordance with certain embodiments of the present invention;

FIG. 6D is an illustrative graphic plot providing an example of a nonlinear separation space determined via radial basis functions;

FIGS. 7A-7B illustrate one configuration of a possible monitor system 78 formed in accordance an exemplary embodiment of the present invention;

FIG. 8 illustrates a confusion matrix showing high accuracy at separating websites by their traffic source when data is encrypted and IP address are hidden from the system;

FIG. 9 is a flow chart illustrating a flow of processes for an inter-user ranging process carried out by a system formed in accordance with an exemplary embodiment of the present invention;

FIG. 10 is a flow chart illustrating a flow of process for a pseudo-code process using all signals identification carried out by a system formed in accordance with an exemplary embodiment of the present invention;

FIG. 11 is a flow chart illustrating a flow of process for a pseudo-code process using blind signal cues carried out by a system formed in accordance with an exemplary embodiment of the present invention;

FIG. 12 is a flow chart illustrating a flow of process for a pseudo-code process using handshaking cues carried out by a system formed in accordance with an exemplary embodiment of the present invention;

FIG. 13 is a flow chart illustrating a flow of process for a pseudo coding process using parsing logged signals carried out by a system formed in accordance with an exemplary embodiment of the present invention;

FIG. 14 is a flow chart illustrating a flow of process for a training procedure carried out by a system formed in accordance with an exemplary embodiment of the present invention;

FIG. 15 is a flow chart illustrating an operational classification procedure of a system formed in accordance with an exemplary embodiment of the present invention;

FIG. 16 is a flow chart illustrating an envelope instrumentation process of a system formed in accordance with an exemplary embodiment of the present invention;

FIG. 17 is a flow chart illustrating a training process of a system formed in accordance with an exemplary embodiment of the present invention; and,

FIG. 18 is a flow chart illustrating an operational classification process of a system formed in accordance with an exemplary embodiment of the present invention.

 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention is directed to the fields of signal processing and data processing, of signature detection, and of computer networks. In the exemplary embodiments disclosed, it is illustratively directed to the blind identification of a website or other Internet Protocol (IP) network connected service or server by characteristics of the IP packet chain other than the content of the data fields of the packets. For example, a system and method implemented in accordance with such embodiments of the present invention may be used to track and identify instances of access to a particular website or web-connected service even when that website is duplicated for counterfeit purposes at different web addresses, is moved, or is hidden behind a proxy service. The system and method address enable the quick and efficient identification of a remote site of interest without the necessity and memory cost of extensively evaluating and comparing actual data payloads of communication signals originating from that site. Suitable applications include but are not limited to:

    • detecting illicit or hostile web services masquerading at different web addresses;
    • detecting malware contacts emanating out of a local network to the internet;
    • rapidly screening and detecting an individual's access to unauthorized sites or site types through a network when such access is masked via encoding or proxy services;
    • detecting predefined types of sessions that are conducted over a secure channels without requiring decoding or other access to payload data; and,
    • detecting use of mobile or desktop applications via their IP transmission patterns.

Briefly, the subject system and method serve toward these and other ends to reliably ascertain the originating sites of communication signals received by a local site over a network, such as the world wide web. The system and method do so in a manner that obviates the need to read or otherwise process the actual contents of the data carried by the signals. In other words, the system and method serve to accurately classify the identity of the remote site which originated a given set of communication signals received in data content agnostic manner.

As used herein, the term “local site” is used primarily for referential purposes. It denotes a reference site with respect to which another “remote” site accessible through a network interconnection may be, operationally at least, remotely disposed. The term “local site” does not necessarily denote a physical site, nor even a site within close physical proximity to a user or other peripheral equipment utilized by the user nonetheless accessing the network through that site. Nor does the term “local site” necessarily mean that a system or method implemented in accordance with the present invention is actually disposed at that site. As described in following paragraphs, such local site may be established physically or virtually using any suitable hardware and/or software measures known in the art.

Remotely disposed but selectively accessible sites of an interconnecting network are typically configured to respond in particular programmed manner when engaged in an interconnection session by another site. In the exemplary case of the internet, as implemented via the world wide web, the remote sites are established as individually accessible websites. When accessed, a website operates much as a server which responds to a client (typically, the local client initiating the access), allocating to the client access to various resources to, for instance, view and navigate through one or more of its web pages. Inasmuch as each web page is typically custom configured in the design of its content, the sequence of responsive actions necessarily undertaken to establish the page for the accessing client is distinctly configured. Such responsive actions typically include the suitable allocation of resources such as text, images, and links to other sites or resources for the accessing client.

In certain applications, the local site may itself be a website or a user platform coupled to the world wide web through a website or other interconnection service providing intermediary. Local sites interconnected by such intermediary include, for example, a mobile communication device programmably configured to access the subject system and method implemented in certain embodiments as an online application executable thereon.

In certain applications, local physical devices may also provide user accessible websites as a means to configure the device or communicate with it. Examples include IP routers and Ethernet enabled cameras which include embedded web server configuration and data exchange pages.

The disclosed example is directed, without limitation, to the identification of websites originating certain communication signals passed through the world wide web. Identification of other web-or network-based transactions may be similarly accomplished in accordance with the present invention, with only minor differences occurring in the nature of the labeling of integrations and signal-subcomponents.

Websites on the world wide web communicate according to one or more communication protocols such as, for example, the widely used Hypertext Transfer Protocol (HTTP). They do so, passing communication signals in bursts, or packets, of data transmitted according to the Transmission Control Protocol/Internet Protocol (TCP/IP), with accessible resources of each website each identified by a unique Universal Resource Identifier (URI). The sequence(s) of responsive actions taken in this context to allocate resources for accessing clients tends to be distinctive for individual websites. In a typical HTTP client-server transaction, the accessed server responds to a client “GET” request for a particular URI with a sequence of data segment transmissions to the client. The response often includes a series of such data segment (or packet) transmission sequences, one for each URI allocated to the client to fulfill a set of related requests within the given HTTP transaction.

For example, where a local site accesses a certain website, a client “GET” request for the URI “/” for a particular web page of the site prompts a sequence of data packet transmissions to establish all or a portion of the web page in question. One or more subsequent “GET” requests may follow to further populate the webpage, for instance, with images or the like identified by corresponding URI's. Those subsequent “GET” requests each entail their own sequence of data packet transmissions suitable for the specific nature of the URI thus allocated. A user at the client, or local site, may then display the web page locally and interact with the originating website therethrough.

Obviously, the need for authentication of the website is of paramount importance, especially where personal, financial, and other sensitive information is shared by a user at a local site with that remote website. While a plethora of authentication and other counter measures are known to combat internet fraud, the growing sophistication of fraudulent sites makes their detection using conventional web page or other data content based measures increasingly difficult. In accordance with certain aspects of the present invention, detection of fraudulent sites is aided enormously by remote site classification measures based on envelope characteristics of communication signals as transmitted by the originating websites.

As used herein, “payload data” refers to the data contents of network traffic, the conveyance of which is the primary purpose of said network traffic. This is distinguished from “envelope data” as used herein, which relates to certain attributes of the network traffic meant to ensure the successful transmission of the payload data to the proper destination. In exemplary applications employing packet transmissions, for instance, envelope data may include a certain part of a packet or packet frame that serves to enable the successful transmission and routing of said packets which carry said payload data.

Preferably, the packets of communications data transmitted by an interconnected website (to a local site) are acquired in time-captured segments which are tabulated according to certain attributes inherently defined in each packet by the communications protocol employed. In the disclosed example, these attributes include such parameters as timestamps, packet length, source and destination address, source and destination port, request method (such as a “GET” command), URI, and the like. One or more of these tabulated attributes may be selected for use as an envelope parameter by which to ‘fingerprint’—or characterize the unique signature of—the website actually originating the locally received communications data packets, irrespective of their payload content. In addition to the directly encoded envelope data, the exemplary embodiment disclosed preferably instruments the timing of the packet stream and records the time at which each packet is captured.

By ascertaining the values of each envelope parameter across the sequentially time-captured data segments may then yield an effective waveform distinguishing the response pattern of one website from that of other websites. Since the web-pages of each website are custom designed and programmably implemented, the response patterns of different websites will vary considerably, although one envelope parameters may provide greater discriminating strength than another. One or more envelope parameters, or combinations thereof, may be suitably selected for use in generating one or more effective waveforms as envelope signals from which a characteristic signature(s) may be derived for the originating website.

In accordance with certain aspects of the present invention, the system and method are implemented to operate in at least a training mode on the one hand, and in a classification mode on another. In the training mode, the characteristic signatures of different websites are generated and stored in this manner. The known characteristic signatures may be updated and refined as more samples of communication signal transmissions in further interconnection sessions with known websites are acquired and processed in the manner disclosed herein. In the classification mode, newly-acquired communication signals may be classified as to their actual website origination, in view of characteristic signatures for known websites. In certain embodiments, these training and classification modes may be switched between rapidly, providing a seamless integration between learning new host sites and verifying against previously visited sites.

As a result of classification, the newly-acquired communication signals may for example be associated with one of the known websites, at which point this fact is preferably confirmed for the local site, and the envelope signals generated for the newly-acquired signals are used to further refine the characteristic signature of that known website. Alternatively, the newly-acquired communication signals may be determined to originate from a website not yet known by a characteristic signature. This fact, too, is preferably communicated to the local site for either detection of a fraudulent site or identification/recordation as another website known by characteristic signature (depending on the application-specific context and circumstances under which the newly-acquired communication signals were received).

Where more than one resource (URI) is allocated to a client during an interconnection session, multiple sequences of time-captured segments are available for use in generating an envelope signal. The envelope parameter values across one sequence of segment time-captured for each resource (URI) are ascertained then to generate an envelope sub-signal (for each envelope parameter/parameter combination selected). The multiple envelope sub-signals resulting from the sequence of time-captured data segments for each URI are concatenated one with the other to collective form the envelope signal pertaining to each selected envelope parameter (or parameter combination). This envelope signal is then used to either newly formulate or refine an existing characteristic signal, or to classify the data segments in actual website origination with respect to characteristic signatures of known websites that have been prestored.

FIGS. 1A-1B and 2A-2B relate to a system 10 implemented in accordance with an exemplary embodiment of the present invention. System 10 provides a methodology which establishes and extracts patterns that occur in IP (“Internet Protocol”) or other data traffic of packetized format during access to a website (including a web service provided through a particular site), and uses such patterns to identify subsequent access to the same. This methodology is independent of both the sending and receiving web site addresses in the transaction which occurs during access, and does not rely on either decoding or understanding of the transaction's data content. Thus, both in establishing a characteristic signature for a website and in subsequent detection of the characteristic signature, the system preserves privacy of data and may operate on any encrypted channels without need for decoding.

System 10 preferably examines patterns of certain envelope information inherent in the packetized data. In accordance with one exemplary embodiment, the envelope information includes a sequence of time-delta values (i.e., inter-packet delay) between data packets or frames of data packets in a captured series. Alternatively, the envelope information includes a sequence of data payload size values over the captured series of packets (or packet frames). As yet another alternative, the envelope information includes a combination of such values over the captured series of packets or packet frames.

An additional advantage of this approach is that the characteristic signature derived from the resulting envelope signals is extremely compact when compared to the captured streams of actual data. Thus, a relatively small amount of data need be stored and analyzed in order to detect and classify a candidate website. This enables far more rapid screening of large bodies of network traffic, with minimal memory and storage requirements even when many different websites are potentially of interest for detection. The subject system and method thus provide numerous advantages over similarly targeted methods that might employ analysis of the data payloads themselves.

The subject system and method also overcome a significant obstacle to identifying websites found in the varying nature of packet exchanges between local and accessed remote sites. For example, due to buffering, a web browser may have downloaded and recorded data segments pertaining to a particular resource (such as for a .jpg resource allocation response) during a prior interconnect session. When encountering a new request for this same data during a later interconnect session, the web browser will likely perform an abbreviated transaction that omits redundant request and download traffic for the buffered item. Exchanges with the same remote website on different occasions may therefore result in transactions that appear to be quite different due to such omission of different portions of the IP packet exchange depending on the circumstances. In accordance with certain aspects of the present invention, the subject system and method account for the potential mismatch of IP packet exchanges between the same local and remote sites by suitably appending partial length IP packet exchanges where abbreviated web transactions are encountered.

Time-Capture of Data Segments

As illustratively shown in FIG. 2A, system 10 at block 110 reduces the web-traffic packets transmitted during an interconnection session (between local and remote sites) to at least a stored record of packet header data. The system captures trains of packets using any suitable means known in the art, such as open source packages like WIRESHARK or the like, which execute to record web packet data in bulk. FIG. 1A illustrates an example of one such chain of packets, a frame of which is time-captured from a network interconnection as data segments delineated in sequence with their various envelope parameter values/indicia tabulated under correspondingly denoted column headings.

To reduce workload, system 10 preferably pre-filters the packets to those of potential interest for the particular intended application. For example, in the website ID application shown, system 10 filters and retains only those frames (i.e., packet sets) for resource requests and responses predicated on http.request.method==“GET.” In the disclosed embodiment, those containing http.request.uri==“/” are subsequently separated from those containing other URI requests, as described in following paragraphs. The packet/frame data retained by system 10 may include, for example such captured parameters as: Time, Packet Size, Source IP, Destination IP, Protocol, Request Method, URI, Next Sequence Number, Acknowledgment Number, Source Port, and Destination Port. Of these data fields, system 10 preferably employs “Packet Size” and “Time” as the envelope parameters most effective for uniquely characterizing a given website. “Packet Number” or any other suitable parameter may be selectively used in alternate embodiments as an envelope parameter of choice. The remaining data fields are referenced in the subsequent parsing of packet sequences into groups for extracting signals.

In the particular list of captured IP packet data segments shown in FIG. 1A, each of the time-captured data segments includes the data fields: packet number, time of receipt, packet length, source IP address, destination IP address, Request Method, URI (where appropriate), Next Seq Number, Ack Number, Source Port, Destination Port, and Host. These fields represent a human readable listing taken from the actual packet stream, and may be encoded in particular fields and format according to any suitable network protocol known in the art.

FIG. 1B schematically illustrates the timing of packet transfers during an interconnect session between a local site (executing a Browser 1, for instance) which as a Client accesses a remote site Server 2 through a network 3. The access occurs according to HTTP, HTTPS, or other web communications protocol. In this particular example, a typical HTML page request is illustrated by the timing and sequence of packet exchanges required to effect and service the request. The browser requests 1a, 1b, 1c are shown in dotted arrows, while the server responses 2a, 2a′, 2b, 2b′, 2b″, 2c, 2c′, 2c″ are shown in solid arrows. Each arrow represents a transmitted packet of communication data.

Note that, in accordance with known web protocols, in some instances multiple packets may be exchanged where information exceeds the limits of a single packet or is divided due to some other protocol rule. Thus, communications data like the index.html data 2a′ may in some cases comprise only one packet, while in others may comprise many packets. The actual number of and labels on packets exchanged shown are simply examples and should not be interpreted to limit this disclosure.

Events shown in FIG. 1B sequence down the page vertically in time, as indicated by the “TIME” arrow 4. Thus, each event appearing further down in the diagram occurs at a time later than the events appearing above it. Activities at either the Browser 1 or the Server 2 are indicated in plain text without arrows around them.

In a typical HTML web page retrieval, as shown here, the first request from the browser 1 is a “GET” of the index page. The “Gee” is a typically used form, and most web servers 2 will interpret the command as equivalent to for example “GET:/index.html,” provided that the referenced file is available. The server 2 then takes responsive action to find and retrieve the referenced file—in this case “index.html”—and transmits the retrieved file back to the browser 1. In this example, a header is transmitted in a first packet 2a, and the body of data in the next packet 2a′. Additional packets (not shown) may be utilized where required by the body of data's length. The Browser 1 then parses the html data, and determines what embedded objects (if any) are required from the server in order to complete local rendering of the page.

In this example, the Browser 1 determines that two embedded images are required. These two files “image1.jpg” and “image99.gif” are therefore requested from the server 2 via packet transmissions 1b, 1c. The server 2 retrieves the first image file and replies with packets corresponding to the header 2b and body of data 2b′, 2b″ for that first image. The server 2 then proceeds to retrieve the second image file, and replies with additional corresponding packets 2c, 2c′, 2c″ to the browser. Again, the number of packets in the response may vary across examples, but will typically be highly consistent for a given website across multiple instances of access.

Note that this requirement may vary, even on a static web page. For example, certain browsers will buffer previously used images. Thus, if the same image is referenced again, while the image is still stored locally, the browser will not request it again from the server. In accordance with certain aspects of the present invention, the subject system and method compensate for partial deletions from a website's expected sequence of data transmission due to such omitted retrieval requests. Otherwise, such gaps in data could significantly distort the envelope signature of the page. Also of note is that many web pages change in their embedded data; for example, selecting user specific information, or inserting or changing advertisements in a page. Such dynamic web pages still tend to exhibit strong partial matches to their prior signatures, however, and are recognized well by the subject system and method.

Each data transmission arrow corresponds to a packet, and a packet payload size is available for each. This information is not shown in FIG. 1B, but form the payload size vector illustrated in FIG. 2B. Likewise, there is a measurable time-delta between each of the packets sent and received. These are illustrated in FIG. 1B (Δti), with corresponding intervals shown. Note that these intervals inherently include the aggregate time elapsed for a number of processes, including the time for each packet to propagate via the interconnecting network, the time for each piece of operative software to parse and act on a data packet, and the time for any retrieval actions, and the like that may need to occur. Hence, the sequence of time-deltas preferably used as an envelope parameter effectively encodes numerous aspects of the server 2 data channel and processing hardware.

The example illustrated in FIG. 1B is deliberately simplified for purposes of explanation. Actual browser host exchanges will generally include substantially more packet exchanges, and may include substantially more transactions within each URI transaction. This is reflected in the example of FIG. 1A, where in response to a single “Get: I” command, numerous protocol transaction exchanges occur even in the absence of any additional URI request. These exchanges include enquiries as to the browser and hardware in use, cookies, security certificates, and numerous other operational details, all of which help form the basis of a website's server/browser interaction signature.

FIG. 1C illustrates a longer portion of the captured data segment example illustrated in FIG. 1A. Note, however, that only HTTP traffic is displayed in this tabulation. In this example, in response to the first “Get: /” request by the browser 1, the server 2 returns not just one or two packets of data, but 27 packets of payload size 1514. Each of these packets has an AckNum value of 2018, indicating they are all in response to the same transaction initiated by the “GET” command with SeqNum 2018. After parsing this HTML data, the browser 1 makes a series of further requests for additional embedded objects, starting with “GET: /default.ashx/id/21589549/”, etc. Each is followed by response packet traffic from the server 2.

In the exemplary embodiment disclosed, only packet traffic that is in HTTP protocol as illustrated in FIG. 1C are preferably used for purposes of establishing signals and signatures—thus ignoring intervening TCP exchanges for those purposes. The traffic is limited to only that between a particular source and destination IP. However, in certain cases it is meaningful to further limit the traffic to just that between a particular source and destination port. In the disclosed embodiment, each signal collection is further limited to only those URI request responses which match the source port of the initiating “GET:/.” Thus, for the data example shown in FIG. 1C, the URI response signals associated with “GET: /id/37631270/” would be omitted, since they are associated with source port “moshebeeri” rather than “gjad816.” In fact, a new request from a separate source port would, in the disclosed embodiment, be used to delineate the beginning of a new combined signal, separate from the initial “GET:/” request.

In certain embodiments, information from all data packets exchanged in aggregate may form the basis of the envelope signal vectors which determine website characteristic signatures. In certain other embodiments, only the server-transmitted packets or only the browser-transmitted packets may serve as a preferred basis of such envelope signal vectors for forming and determining the characteristic signatures.

In collecting signals, the preferred embodiment takes the time delta vector T from only the actual packets linked into a URI transaction, after other packets have been filtered out of consideration. This aspect is illustrated in FIG. 2B by the times labeled Δt5b and Δt8b. The sub-signal Tim1 associated with “GET: /pics/image1.jpg” in that example will take as its first values Tim1=(Δt5b, Δt6, Δt7, . . . ), while the sub-signal associated with “GET: /pics/image99.gif” in that example will take as its first values Tim99=(Δt8b, Δt9, Δt10, . . . ). This contrasts with other embodiments, which may instead consider the overall sequence T=(Δt5, Δt6, Δte, . . . ) for purposes of forming envelope signals and signatures.

Referring back to FIG. 1B, the order of packet exchanges may vary depending on the particular application. For example, the Browser 1 may make multiple “GET” requests before the server 2 responds. In that case, the order in which the “GET” requests are served may vary depending on inherent aspects of the Server 2. In other cases, a particular browser 1 may wait for a reply before sending its next “GET” request, as some browsers limit the number of concurrent outstanding requests they permit.

An envelope sub-signal, with reference to FIG. 1B, preferably comprises information derived from one “GET” request and its corresponding replies. Even where envelope sub-signal packets overlap in real-time, they may be parsed and assembled using the NexSeqNum and AckNum fields shown in the captured data segments in FIG. 1A. These reflect a transaction identifier code embedded in each packet intended to identify request and reply packets associated with a transaction. Thus, in certain embodiments of the invention, allowances are preferably made for out-of-order replies, accordingly assembling each sub-signal in view the transaction identifier code.

Moreover, in certain embodiments, the “GET” command associated with each envelope sub-signal may be recorded for reference purposes, so that envelope sub-signals for future transactions may be matched for efficiency to the previously recorded envelope sub-signals. Suitable gaps may also be left where a particular request fails to occur and/or the envelope sub-signals re-ordered to correspond with the order indentified when an earlier pre-recorded occurrence of the envelope sub-signal was ascertained.

Note that, at a simplified logical level, the Browser 1 software interacts with the Server 2 software, regardless of the intervening layers (which are not shown). In practice, each browser 1 is typically embedded in a hardware platform, such as a desktop PC, a tablet, a mobile smartphone, or other device. This hardware platform is linked via a local router to a local network. The server 2 software also operates on corresponding server 2 hardware, which again may be any type of physical device. The server 2 may comprise any suitable platform known in the art, ranging from an embedded device interface to a full commercial internet server, including multiple load-balanced instances of the served site. The local network may be linked via a gateway to the world wide web (internet) and thence to the server; or, the server may be disposed on the same local network. There may be a firewall, proxy servers, and any number of complex routings, all of which are encompassed by the term “network” and its variants, as used herein. Such intervening layers may affect the absolute measured timing values used in forming an envelope sub-signal, envelope signal, or signature, but will not materially affect execution of the disclosed system and method, so long as they remain consistently in the path of a data exchange. That is, the precise nature of the data path is immaterial to the operation of the subject system and method.

Still, it is notable in this regard that the network cloud between the two devices may be quite complex. It may include secure tunneling or other layered protocols. The network may also route packets belonging to other web sessions in a manner that overlaps and interleaves with a concurrent web session under consideration. In accordance with certain aspects of the present invention, the disclosed embodiment preferably includes suitable measures to separate one web session from another, in addition to those for extracting envelope signal (signature) information from the captured packet exchanges.

Packet captures may be done on a continuous basis, or in bulk. In the latter case, recordings may often reach millions of packets. Thus, in certain embodiments it will be understood that use of a continuous rolling buffer(s) may be more practical and efficient in the handling of such data. It is typically necessary to reduce the set of captured packets to those salient to the detection process required for the intended application. In the disclosed example which specifically targets website identification (ID), the salient packets are those relating to the sets of GET request initiated from a local site, and the sets of responses to such requests transmitted by a remote site thereby accessed during an interconnection session. The Source and Destination IP address fields of the data segments reflect the bidirectionality of the interaction in a typical session.

Returning to FIG. 1A, the interaction of interest begins in this example with the “GET” command from the local site address (69.244.66.17), to which the remote server (addressed at 63.69.72.43) responds, the local acknowledges, etc. In the illustrated example, TCP protocol packets are part of this interaction interspersed with the HTTP protocol packets the website sends from its server. Taken together either for a fixed time period, or until the session is interrupted, this stream of packets serves as the basis of the envelope signals as described in preceding paragraphs.

In certain embodiments, data segments of only one protocol type may be used to establish signals, while in other embodiments multiple protocols may be leveraged. In preferred embodiments, data segments of all protocols that partake in a given IP packet exchange relevant for detection are included in the captured data. As discussed in following paragraphs, application of the disclosed exemplary embodiment to HTTP website ID applications may discard certain packets from consideration in order to establish a more robust envelope signal to signature comparison, for example relying only on HTTP protocol packets, and only on those which flow between specific source and destination ports.

General System Operation

FIG. 2a illustrates the flow of processes within system 10 in schematic form. At blocks 120, 122, and 124, system 10 parses the time-captured data segments to obtain one or more them into URI-labeled envelope sub-signals which may then be concatenated to generate a combined envelope signal for an envelope parameter. To do this, all “GET” commands are preferably identified in the illustrated example and used to specify the starting point of a signal of interest. They are preferably label by Destination IP Address and URI. Typically (but not always), the initial GET operation preferably but not necessarily refers to the website home page using the “1” URI label. Other initial queries may be keyed upon in like fashion. For example, it may be established from timing and sequence codes that any given browser-side GET operation to a URI address is independent of previous GET operations and therefore constitutes the start of a new interaction session initiated by the user or by software executing on a user machine. In alternate embodiments where network processes are not HTML (HyperText Markup Language) website-based may be accommodated by adapting the system and method exemplified herein to the known base interaction characteristics and protocols of the network process of interest. In certain embodiments, the interaction process and the start keys may be learned by example and training.

Once the “GET” command data segments are identified, the other captured data segments are scanned for correspondence in Acknowledgment (Ack) Number, IP Addresses, and Ports to identify the data segments for packets constituting the remote site's response to a particular “GET” command. The envelope data properties (e.g., time and payload size) of interest are then gathered to generate a URI-tagged sub-signal for each response following a “GET” command. The “Parsed URI Signals” which result are indicated at block 124.

In the illustrated example, the “GET” command at packet number “2176436” represents the initiation of a resource allocation request, and the subsequently listed data segments represent the sequence of programmed actions taken by the accessed remote site in response. The sequence of these subsequent data segments comprise the source data from which an envelope “signal” (or sub-signal where data segments from multiple resource allocation requests are captured) is derived for use in the system's signature and identification process. For example, the sequence of time values (or the sequence of differences between time values) acquired across successively listed data segments following the “GET” command comprises one discriminating vector that may serve as an envelope signal or sub-signal. Likewise, the sequence of packet length values acquired across these same data segments comprises another such discriminating vector. Each sequence of parameter values thus obtained along a corresponding data field column across the given set of successively listed data segments then provides a time series of data which may be illustrated as a waveform.

The envelope sub-signal formed in this manner by system 10 for the resource allocation response (by the interconnected remote site) to the local site's “GET” request for the first URI “/” is indicated at block 126. In the illustrated example, data segments similarly captured for other resource allocation requests are available (though not shown in FIG. 1A; see description of FIG. 1C). These other resource allocation requests may be for resources such as image files identified by other URI's (such as “.jpg” or the like) or other embedded resources required for populating the browser display of a certain web page of the remote site. Additional envelope sub-signals formed based on the data segments captured in relation to these other resource allocation requests are indicated at block 128.

Thereafter, an attempt is made to combine all URI-tagged sub-signals (for the same envelope parameter) from the same IP Address in a given interaction into one composite envelope signal. To do this in the disclosed embodiment, the “GET” “/” commands serves as the composite envelope signals' start point. Each URI-tagged sub-signal is then matched to these signal start points and check for sequence consistency, as further described in following paragraphs. The consistent URI-tagged sub-signals are then concatenated together to obtain at block 130 the Combined (Envelope) Signal for the current IP Address and GET “/” operation instance. Other user initiated GET URI requests may replace the GET “/” at block 126 as an indicator of the start point of the Combined Envelope Signal without material change in the functionality of system 10.

The combined envelope signals so obtained are then saved at block 132. System 10 may be operated in a training mode, whereby such envelope signals are obtained over a plurality of interactions between a local site and a remote site. The envelope signals generated from the repeated interactions may be accumulated as samples for the given local-remote site combination at storage block 132.

In order to collapse across samples in a training corpus thus acquired for each local-remote site combination, predetermined analytical measures are applied at block 134 across the envelope signal sample set to obtain a “typical” envelope signal vector representative of the remote website at hand. Preferably, a simple mean of the sample set is taken in this regard. Alternatively, other analytical measures for obtaining a composite/representative envelope signal may be used in place of a simple mean operation. The choice of suitable analytical measure will depend on the particular requirements of the intended application. The typical envelope signal is then taken to be the “Website Signature” of the remote website, as indicated at block 136. One or more website signatures may be thereby acquired for each of a plurality of remote websites, making them ‘known’ websites. The website signatures may be stored in a signature database 138 for later retrieval, so that the original training signals need not be immediately present when classifying newly-acquired signals for website origination in a subsequent classification mode.

As illustrated, to compare a derived or stored website signature with a newly-acquired, or novel, signal received at block 140, a similarity test comparison between the novel signal and a Website Signature is conducted using a Comparison Metric to obtain a corresponding Comparison Value at block 142. The comparison metric in one examplary embodiment is preferably an L-2 norm (root mean squared vector distance); however, numerous other metrics known in the art may be used in alternate embodiments. Such other comparison metrics include but are not limited to other norms such as L1 and the like; hamming distances; vector angles; and, matched filter processes. Other comparison metrics may involve more complex analyses, such as those employing principle component and sparse decomposition methods (as further discussed in following paragraphs). The Comparison Value is typically one scalar value for each of the dimensions of comparison, such as a distance.

In certain applications, multiple website signatures based on different dimensions of comparison (different envelope parameters) may be generated for each website. These multiple website signatures may be compared to the envelope signals obtained for the novel signal data according to corresponding envelope parameters. This enables multiple dimensions of comparison for an individual instance of remote site access.

Distance from any one of the multiple website signatures may be the basis of a decision point when comparing novel signals to the known set. In certain applications, website signatures according to multiple envelope parameters (feature types such as packet length and packet time) may be used for joint assessment of novel signal similarity to the signatures of a known website of interest.

In the exemplary embodiment disclosed, a pairwise comparison of features is implemented using linear support vector machine (L-SVM) type decision processes to assess them jointly by segregating areas of a 2-dimensional plane. Decision processes in certain alternate embodiments may be based on a mathematically combined set of comparison values, a voting process, a decision tree, or the like. Other higher dimensional decision processes based on more than two features (signal vector types) may also be employed, as those skilled in the art will recognize in view of the disclosures herein.

FIG. 2B illustrates examples of parsed URI tagged signals obtained from a set of time-captured data segments for a particular packet sequence much as illustrated in FIG. 1A. The parsed URI signals are labeled with the URI of the originating command and the IP Address, and represented (in this particular example) by three column vectors for packet time, inter-packet timing, and payload size envelope data (Data1, Data2, Data3) extracted across the sequential listing of data segments. Other envelope data may be extracted in other embodiments, and these three illustrative vectors are not intended to be a limiting example. For instance, portions of the payload data might also be used; however, this would largely forfeit the gains in speed, flexibility, and computational burden mentioned in preceding paragraphs. Apparent packet size may also, for example, be measured and used in place of Payload size, fully decoupling the instrumentation of the packet envelopes from their transmitted header data values.

Because the vectors Data1, Data2, Data2 shown in FIG. 2B are time-series vectors (kept organized in a linear fashion as described in following paragraphs), numerous time-series analytical processes may be applied to isolate and identify websites by their overall features. This may not be appropriate on any particular component of the actual payload data. Moreover, the time-series are of relatively small size, even where the total payload data content is significant. Since the envelope sub-signals are defined by these time-series vectors, and since the website signatures are ultimately derived from such sub-signals, each website may be identified by a much more compact signature than would be the case if it were identified by its payload data.

FIGS. 2C-2D are flow chart illustrating the flow of processes in a method carried out by use of system 10 or the like in accordance an exemplary embodiment of the present invention. The flow chart of FIGS. 2C-2D are similar, except that FIG. 2C reflects the particular use of delta time and payload size as envelope parameters (for the feature vectors forming the envelope sub-signals), whereas FIG. 2D reflects the more general use of any extracted feature vector sequence to form one or more envelope sub-signals. Like reference numbers are therefore used in the interests of simplicity and clarity to indicate similar steps in the two flow charts, although minor notational variations consistent with the respective embodiments represented are reflected in certain steps.

At block 21 in these embodiments, the collection of captured packets is parsed by user browser session. This is preferably done using information about the source and destination IP addresses, in any suitable manner known in the art. At block 22, sequence and acknowledge codes within the captured packets are used to group the packets by transaction (or resource allocation sequence). Each group of packets for a transaction is labeled by the initiating packet in the transaction sequence, typically a URI “GET” command in the shown examples as applied to HTML web page identification.

At block 23, selected sequential feature vectors are extracted for each of the URI transactions to form envelope sub-signals. In the primary example shown in FIG. 2C, a vector P is selectively formed by extracting the payload data size values for each of the packet data segments within the grouped segments defining the given URI transaction, and a vector T of time delta values between successive packet data segments is selectively formed by extracting the same from the grouped segments. As indicated at block 23 in FIG. 2D, other embodiments may such feature vectors based on one or more envelope parameters selected as the vectored feature(s).

Beginning at block 24, the envelope signal vectors for each initiated new host access are assembled and compared, in order to verify that host (remote site server) against past activity. The “user” may be a human user, or an automated user programmably established in software, but in any case it is independent of other URI transaction sequences and therefore recognized as a new request for access to a website and signal.

The flow then proceeds to block 25, where the URI envelope sub-signals previously learned from past access to this host website is retrieved. This step may be optional, as the subject system and method in certain embodiments will assemble the envelope sub-signals blindly, in the order that packet data segments are received. In many instances, such blind sequencing is effective since website interactions tend to be relatively consistent; however, in the preferred embodiments, URI orders are recorded so as to better compensate for browser pre-loads and other website interaction dynamics while avoiding false positive detect conditions.

At block 26, the URI envelope sub-signal sequences are assembled either sequentially, or if the information referenced in connection with block 2-5 is available, according to the order reflected in previously recorded envelope sub-signals for the given URI. In certain embodiments, any gap in a combined envelope signal are filled with a suitable number of zero value entries for URI envelope sub-signals not present in a particular instance but present in a past recorded instance. As a result, the similarity metric employed is minimally affected by the omissions. The assembled sequence of URI envelope sub-signals yields a combined envelope signal, or simply “Combined Signal,” as used herein.

At block 27, envelope signal vectors for previous instances of the given host website are retrieved. The retrieved envelope signal vectors may include a mean signature vector, a sparsely reduced signature vector, or in certain embodiments, a collection of past recorded samples, each of which forms a signature vector reference. Having made the retrieval, a metric comparison is formed at block 28 between the newly obtained Combined Signal (feature vector) and the retrieved signature signal vector(s). Each comparison between a new feature vector and a previous signature feature vector is preferably reduced to a numeric value. Depending on the particular requirements, different embodiments in this regard calculate such values as an L1-norm of the difference, an L2-norm of the vector difference, a vector inner product, a vector angle, an RMS difference, or the like. These examples are given without limitation, noting the objective of reducing the comparison between two signal vectors to a scalar value, which objective may be realized by numerous measures known in the art. Preferably, an L2-norm of the vector difference is calculated in the illustrated examples.

At block 29, the resulting scalar value or values are place into a suitable context for quick and simple comparison. In the two feature example employing feature vectors P and T, each vector is reduced to a single scalar comparison value, which may then be placed on a graphic plot within a two dimensional plane similar to that illustrated in FIGS. 5A-5B. The location of the plotted point relative to the previously determined dividing line (50) enables the determination at block 31 as to whether the host website access event in question is consistent with a host website for any previously learned website signatures (as indicated at block 32) or not (as indicated at block 33). If not, an alarm is actuated at block 33 to inform the user, to make a log record, or to otherwise handle the exception.

If operating with only one value in one feature, it may be sufficient at block 31 to make a simple thresholding or other check for verification/non-verification of remote website host identity. Such a threshold is illustrated in the example of FIG. 5B for the instance where Payload size or the Delta-Time metrics axes are independently considered and used to distinguish most of the True and False points of verification.

If operating with more than two envelope features, then several methods of combining information known in the art may be suitably employed. An SVM may be trained and evaluated in more than two dimensions. Alternatively, the SVM may be trained and evaluated in pair-wise combinations of derived metrics, and a voting scheme used to determine whether or not a false host site has been accessed.

FIG. 3 illustrates examples of parsed signals as time-series graphic waveform plots. In this example, three separate instances of three URI-tagged envelope sub-signals are generated for data segments captured for corresponding resource allocation responses of a particular remote website. Note that different instances (labeled 1, 2, and 3) of the same URI-tagged sub-signal are of different lengths (due for instance to buffering of data acquired from prior interactions with the given website). The sub-signals obtained for the different URI are concatenated into one combined envelope signal (1-3) for each of three website visitation instances.

Zero padding 30 is incorporated to normalize the relative vector lengths where URI-tagged sub-signals are missing or are shorter than their peers in other instances. That is, the abbreviated URI tagged sub-signal is appended with zero values to match in length the longest URI sub-signal peer (of other captured instances). The padded sub-signal is then concatenated with the other URI-labeled sub-signals of the same interaction instance to obtain the Combined Signal for that instance. Preferably, a check is made for consistent signal count. Checks are also preferably made to ensure that each URI signal event matches only one start signal (i.e., URI-tagged sub-signals where the URI is “/” in the working example), and that each start signal is matched by only one URI signal of the specific URI. In this manner, envelope information ascertained from streams of packet transmissions is combined to form a time-sequence signal that may be used to uniquely characterize a website.

FIG. 4A illustrates another, more detailed flow diagram of the training and test (or classification) modes of operation preferably undertaken by system 10. To obtain a working classifier system 10 is first trained to recognize certain known websites by their characteristic signatures using training signal samples. These training signal samples include “True” training signals indicated at block 40 that correspond to the target website of interest. The comparison values of these “True” signals are found to often form a fairly tight group in a graphic distribution. Thus, in many applications, this will provide sufficient training, and a simple distance metric is applied to each of the feature signature signals. In such an embodiment, a novel signal is determined to match the training set when its comparison value lies within a pre-established distance of the training group; otherwise, it is determined to not match. However, to ensure sufficient reliability in other more challenging cases, the training signal samples preferably also include in the disclosed embodiment a set of “False” training signals indicated at block 42 that correspond to websites other than the target website. This provides a basis of distinction for the target website from other sites, and corresponding guidance as to how much variance one might expect in instances of true-positive hits as compared to a sample set of potential false-positives. A comparison metric value is generated at block 43 for each signature feature of the target website and subjected to SVM training over multiple instances of interactive access to obtain a set of Website Separation Parameters at block 46 which provide negative comparison measures for disassociating a novel signal from the target website. This multiple instance training is typically but not always the case, as single exposure training/learning may suffice in certain applications.

The comparison metric may take many forms. In the illustrated example, an L2-norm metric is used. For each element in a mean envelope signal vector obtained for the False Training Signal, a difference from the corresponding element in the characteristic signature mean vector is taken then squared, after which the squares are summed and a square-root taken. In two-dimensions, this is simply the Euclidian distance between the test signal vector and the corresponding mean of the given signature vectors. The distances of the True signals' mean envelope signal from the mean signature signal are taken as well to populate a graphic plot with points for each instance of access to the website.

This is illustrated in FIGS. 5A-5B (described in more detail in following paragraphs) which show graphic plots of the comparison values obtained at block 43 (of FIG. 4A) based on inter-packet time and payload size comparison metrics. A dividing line 50 established between the regions of True and False training signals provides decision criteria by which subsequently acquired novel signals may be rapidly classified against this target website.

FIG. 4B is a flow chart illustrating the flow of training processes learning signatures from new websites in a method carried out by use of system 10 or the like in accordance an exemplary embodiment of the present invention. At block 4a, the collection of captured packets is parsed by user browser session. This is preferably done using information about the source and destination IP addresses, in any suitable manner known in the art. At block 4b, sequence and acknowledge codes within the captured packets are used to group the packets by transaction (or resource allocation sequence). Each group of packets for a transaction is labeled by the initiating packet in the transaction sequence, typically a URI “GET” command in the shown examples as applied to HTML web page identification.

At block 4c, selected sequential feature vectors are extracted for each of the URI transactions to form envelope sub-signals. In the primary example shown, a vector P is selectively formed by extracting the payload data size values for each of the packet data segments within the grouped segments defining the given URI transaction, and a vector T of time delta values between successive packet data segments is selectively formed by extracting the same from the grouped segments. Other embodiments may such feature vectors based on one or more envelope parameters selected as the vectored feature(s).

Beginning at block 4d, a training loop is carried out for each independent website host of interest, to be referenced as known remote sites for future detection and classification. The remaining steps carried out at blocks 4e-4k occur for each website host of interest. These steps of blocks 4e-4f are shown and described in the context of only one exposure to a host, but will preferably operate on a collection of numerous exposures to the same host. Training may be carried out one time on a collection of examples, or in certain embodiments, the training may be carried out iteratively, adding each confirmed access to a given host into a pool and retraining frequently. The latter approach is more adaptive, and provides for learning on a continual basis from ongoing website exposure.

At block 4e, the URI Sub Signals extracted at block 4c are recorded along their order of occurrence. After a predetermined number of samples are captured, they are used to train the method. Block 4f begins by determining the typical order of URI transactions. The captured samples are searched to determine the maximum URI transactions that have previously been associated with the given host, and the first is chosen if more than one past transactions are found of similarly long length. At block 4g, each of the other training samples is then ordered to match the longest sample on record, leaving gaps where URI transactions have been omitted. For storage purposes, any filler code may be used in the gaps. For mathematical purposes, the filler value is preferably chosen to create minimal effect on the resulting signature. Thus, zero values are found to be appropriate in the disclosed example.

Each collection of URI transaction sub-signals is concatenated into one long combined envelope signal representative of the host, and each resulting envelope signal vector is preferably of the same length. A combined envelope signal vector is formed for each envelope feature of interest. Consequently, this reduction to a combined envelope signal vector is made in the disclosed example for each vector P of payload data sizes and a vector T of time deltas between packets. In other embodiments, such reduction to a combined envelope signal vector is made for each of these and/or any other selected feature (envelope parameter).

At block 4h, a typical signal vector is formed, which comprises a representative signature of the web host. In one embodiment, a mean of the envelope signal vectors collected over different sessions is taken. In other embodiments, a root mean square (RMS) average is taken over the envelope signal vectors. In yet other embodiments, a sparse approximation of the signal vectors is made and equivalent means are found in the reduced dimensional space which results. A typical signal vector is thereby formed using such measures for each combined feature vector. In the given example, a typical signal vector is found for each vector P of payload data sizes and a vector T of time deltas between packets.

Once a reference signature has been established for the “true” training group, in the disclosed embodiment, a random collection of “false” training samples are selected at block 4i from host sessions known to be other than with the true target host. These may include host session traffic with all other recorded hosts to date, or may include simply a random selection of web host session traffic.

At block 4j, each recorded host session's combined envelope signals—that is, all known true training samples and all known false training samples—are taken to form a metric comparison for each with respect to the typical signature vector(s) determined at block 4h. Each comparison between a training feature vector and a signature vector for that feature is preferably reduced to a numeric value. Depending on the particular requirements, different embodiments in this regard calculate such values as an L1-norm of the difference, an L2-norm of the difference, a vector inner product, a vector angle, an RMS difference, or the like. These examples are given without limitation, noting the objective of reducing the comparison between two signal vectors to a scalar value, which objective may be realized by numerous measures known in the art. Preferably, an L2-norm of the vector difference is calculated in the illustrated examples.

SVM training is then performed at block 4k on the resulting values, with each value plotted into an appropriate dimensional space, and a separation divider (or partition) is determined therefrom. The divider provides a delineating reference by which values obtained from future host sessions may be classified.

Other Comparison Methods

It will be apparent that SVM is not the only comparison and classification tool that may be applied. While an SVM approach is preferred, those skilled in the art will recognize that once a web host access instance has been reduced to a signal vector as disclosed herein, other known approaches may be suitably taken. For example, a fixed threshold to any comparison metric disclose herein may be applied in order to determine whether a subsequent access to the same apparent web host is or is not “similar” enough to previous instances to constitute a match.

It will also be apparent in light of the disclosed system and method that one need not explicitly reduce each web host to precisely one combined envelope signal for each feature type (envelope parameter). For example, specific envelope sub-signals may be used individually, or combined into subsets, in order to form the basis for comparison.

Moreover, in certain embodiments, resulting URI sub-signals need not be concatenated into a combined signal. Each URI sub-signal may be considered alone for purposes of training and testing, and omissions in any web-host access instance simply ignored in the comparison. In certain embodiments, detection of each URI sub-signal may be made independently by an SVM classifier or another detector, and the combined results of multiple detections used to determine the likelihood that a particular host has been accessed. It should be noted, however, that the exemplary embodiment disclosed which reduces each web instance to one signal and includes Δt information will benefit from encoding the timing between typical URI “GET” signals, as this information is lost when URI sub-signals are treated independently.

Referring back to FIG. 4A, a process identical to that described for sample training signals is preferably employed during a classification mode of operation for system 10. Envelope signals obtained for a novel signal received at block 44 are compared to the mean True training signatures by a comparison metric to obtain Comparison values 45 for each signature feature.

By placing this point in the exemplary embodiment disclosed within the context of the space defined by suitable SVM training it can be determined whether the Novel Signal is more similar to the target website or more similar to that of the false training samples and therefore distinguishable from the target website. In alternate embodiments not illustrated, means other than SVM may be used. For example, a single comparison value, a threshold value, or the like may be used as to test similarity of the Novel signal to the signature signal of any target website of interest. A novel signal may then be quickly and effectively evaluated for similarity to multiple target websites of interest simply by repeating this process using other training data. Note that, the disclosed embodiment simply requires that Feature Signatures and Website Separation Parameters be available in order to make a comparison. Thus very little information need be retained on hand to enable a fast, reliable detector that can be used for screening large amounts of network traffic for relevant hits.

FIG. 5 illustrates example planer SVM separation of Data. SVM is a technique known in the art of machine-learning. As used herein, the SVM is directed to a computer implemented process that attempts to calculate a separating partition between different categories of data. The data is projected into a plurality of dimensions, and the partition will comprise a surface in a dimension less than that of the projection. Thus, in certain exemplary applications, data is projected in two dimensions, and a line comprises the partition. In three dimensions, the separating surface would comprise a plane; and, in N-dimensions, the separating surface would comprise a mathematical hyper-plane. Without loss of generality, it is possible to use curved surfaces in place of a linear surface for the partition.

In general, the partition effectively separates the data-space into two ‘half’ spaces, corresponding to the categories of interest. It is feasible to segment the space into more than two regions where necessary in other embodiments and applications. Linear surfaces and bi-section are preferably used for computational speed. Depending on the application, a voting system may be constructed that enables multi-featured data to be addressed deterministically.

In FIG. 5A, the points 52 at the bottom left region of the plot represent True training signal samples for the target website of interest, while those points 53 at the upper right region of the plot represent the False training signal samples known to correspond to other websites not the target. Each point marked with a cross represents one test case (an instance of website access), placed into a two-dimensional planner space by comparison of its respective Time metric distance and Payload Size distance from the group mean signatures based on each feature. In FIG. 5B, the dividing line 50 represents a natural separation of the space between the True and False samples. The line 50 is obtained by a suitable SVM process known in the art. When envelope signals generated for a novel signal is compared to the mean group signature for the target website in the two dimensions illustrated, they may also be placed in the 2D plane shown. If the placement is to the upper right of the dividing line 50, the novel signal is classified as originating from a site different from the target website. Otherwise, if the placement is below and left of the dividing line 50, the novel signal is classified as originating from a site very similar to the target website and therefore likely to have originated from the same site (as indicated at block 48 in FIG. 4A).

In this example, each of the points disposed in the lower left region of the plot which represent different instances of access to the same site are neatly clustered away from the points of other websites. This illustrates how readily websites may be distinguished from their peers based on certain envelope parameters without the need to read, decode, or otherwise use IP address or payload data content to do so.

To illustrate an alternate embodiment, FIG. 5B also includes a point 55 disposed on line 50. This point 55 represents a threshold by which a single dimension of comparison may be used to distinguish the target website from the majority of its peers. Note that any individual dimension illustrated in this example makes for an imperfect separation, while the 2D application of both feature dimensions in the plane make for a perfect (or complete) separation of the target website from its peers.

FIGS. 6A-6B further illustrate planer SVM separation of Data tested against a larger dataset. In the illustrated case, a different IP address is used to establish a target website signature. FIG. 6A shows the entire space of comparison between the target website and over 6000 others. As it is difficult to see the cluster of true-positive points in this plot, due to the other sites' extremely large range of divergence from the target, FIG. 6B shows an enlarged view of a lower left corner of the plot. Again, those points to the lower left of the dividing line 60 comprise samples of signals extracted from the target website while those to the upper right of the line 60 comprise samples of signals extracted from all other websites. Perfect separation is achieved by the dividing line 60, even with a large number of test cases.

Parsing

Turning more closely to the unit within system 10 for carrying out the parsing of the captured segments of packet data to signal data, the first task in the parsing process is to identify which points in the packet chain represent the signal starting positions. In the disclosed embodiment, the signal start positions are defined as frames with a Protocol of “HTTP” and a Request Method of “GET.” In the particular example shown in FIG. 1A, a Source IP Address of “69.244.66.17” is used. The recording has been made on the outside of a NAT firewall and thus all request data of interest will appear to be sourced from this IP address. To identify all members of the same signal, the corresponding packets must be linked. The Source and Destination IP addresses, Source Port, and Next Sequence Number are saved for the start signal. These are then compared to the Destination and Source IP addresses, Destination Port, and Acknowledgment Number of all potential members of the signal in the captured data chain. By checking IP addresses, the members of a signal are restricted to those coming from the same address, and by checking the Port and Acknowledgment Numbers, the members' response to the previously identified “GET” request is ensured. Signal types are labeled based on the Destination IP address of the start signal and the URI (“universal resource identifier”) of the start signal (see FIG. 2B). A “signal” is thus considered to be a data column vector obtained from all packets returning information from the same website during the same brief session of interaction. In this example, an interconnect “session” may be limited by either a maximum timeout duration or by the occurrence of the next “GET” “/” request, whichever occurs first.

In the exemplary embodiment disclosed, the Payload Size of the member signal and the Time at which the member signal was recorded are extracted. In addition, the capture Time of the start signal is recorded as the first point of the time signal. This enables use of inter-packet Time (the sequential differences in arrival times between packets) as a signal of equal length to the Payload Size signal.

Concatenation

A website typically references different component files indexed by their URI to display itself (e.g. images are saved separately and put together upon viewing with the HTML code). In order to retrieve this extra information, an accessing computer will send a “GET” command for each bit of extra information. Each additional “GET” operation results in an additional envelope sub-signal with a new URI label that may be linked to the starting sub-signal resulting from the initial “GET” operation. Thus, all of the extra information sub-signals are preferably gathered with different labels and appended to the starting signal in order to obtain a combined signal representative of the entire website download process.

The first step in the disclosed embodiment is to identify all signals with the same IP address. Next, all of the “GET” “/” commands are identified and set as the starting points for the combined signals to be concatenated. The URI sub-signal associated with the starting point “GET” “/” request is termed the “start signal.” Also, the time that all “GET” commands were made are retained, so that time windowing may be used to determine which concatenated signal an extension belongs to. Preferably, only URIs that match at least a majority of the start signals are added (this allows some margin for internet errors such as HTTP 404 errors), eliminating from consideration any URI's not having enough sub-signals.

Next, each URI-labeled sub-signal (Parsed Signal) is examined to attempt a match of the signals they contain to the existing start points for the concatenated signals. To do so, the difference in times between all “GET” “/” commands and each other “GET” <URI> commands are calculated, and each URI sub-signal is then mapped to the appropriate start signal. All URI commands are mapped to the start command signal (“GET” “/”) to which they are closest, with those URI sub-signals that came before the closest start signal then eliminated from the group. Checks are then made to ensure that only one start signal exists for every URI command signal and that a majority of the start signals are actually matched with a URI sub-signal. The typical sequence in which URI sub-signals follow their respective start signal is then determined from the example set. If a URI sub-signal in a particular position in the typical sequence is not matched in a majority of the training cases, then the URI sub-signal is determined to be spurious and eliminated from the signature pattern. If a start signal contains more than one mapped sub-signal with the same URI label in the same or similar sequence position, the entire command sequence is eliminated from the training signal list. This may occur, for example, where users navigate to a particular website component URI themselves, thereby circumventing the normal “GET” “/” command and casting doubt over the matching process.

The URI sub-signals are then concatenated to the start signals to form a combined signal representative of each website access instance. Note that the individual envelope sub-signals may be different in length from other sub-signals from the same URI request command. As noted previously, this may occur because of dynamics in the download process such as buffering of images by a browser program or the like that reduce the requisite amount of data transfer. Thus, longer and shorter start signals and URI command sub signals may result. To compensate for this, a string of empty-value codes are preferably appended to sub-signals of each URI abbreviated in this regard, so that an identical number of elements make up each URI command's combined envelope signal. This includes the start signals. It does not matter if different signals for different websites are of different lengths, only that multiple signals for the same web site are of the same total length.

The disclosed embodiment allows one to limit the combination signals based on time between commands, by forcing URI commands to be within some time threshold of the start signals. This provides an additional cross check that the URI commands are actually occurring subsequent to and responsive to the same “GET /” (or other) start signal.

When executed, the disclosed embodiment creates one set of signals for each “GET” “/” command in the data, whereby an envelope “signal” is created from a set of captured packets. A substantially identical process is carried out both for collecting training data and for collecting novel signals which are to be classified.

Classification

Once a full set of website signatures is obtained, newly-acquired communication signals may be quickly and conveniently classified according to their remote site origins. That is, the envelope signal(s) obtained for the newly-acquired communication signals may be compared to known website signatures for classification in this regard. Before the envelope and signature signals may be compared directly, they must be aligned in length. The comparison processing thus accounts for the presence of empty-value codes properly which may have been introduced during the combining of URI envelope sub-signals as described in preceding paragraphs. Where necessary, the padding value are changed to zero values, and any additional padding required is incorporated by adding zero values to accordingly lengthen all signals from the shorter set. This forces the two signal sets for comparison to be of the same length.

In the exemplary embodiment disclosed, a test signature is formed by taking an average (mean) across all signals from the training group. This is the most basic training mode of operation. More elaborate weighted averages may be used in other embodiments, as may more complex analytical approaches such as principal component analysis to reduce the dimensionality of the signals.

One analytical approach of particular note in this regard in other alternate embodiment is the execution of a simultaneous sparse approximation to reduce the dimensionality and concentrate information in only a few terms (such as disclosed in U.S. Pat. No. 7,079,986 entitles “Greedy Adaptive Signature Discrimination System and Method,” issued to J. Sieracki). An advantage of combining the use of such simultaneous sparse approximation methods with SVM is that high-accuracy decisions may be made thereby based on a sub-space of only two dimensions—which further reduces computational complexity. In these and other applications, algorithmic measures for calculating a partition line are not limited; and, any fast approximating algorithm may be employed for a partition even if that algorithm works only in two dimensions.

Returning to the exemplary embodiment at hand, any instance of a website visit may then be reduced to a comparison value. Preferably, an L-2 norm (root-mean square distance) is applied as described in preceding paragraphs between the average training signal and individual testing signals of same type (classification dimension, envelope parameter). This results in two values for each training signal (in the particular case illustration): a Payload Size Comparison value and a Time Comparison value. These can be placed as points in a 2D plane. The final determination of classification is obtained preferably via an SVM classifier as described in preceding paragraphs.

In this manner, it may be determined whether or not any particular derived signal is derived from a target website (whose typical website data stream characteristic has been learned), or is in fact derived from a different website. In similar fashion, detection of remote site origination in any other network based interaction may be determined without recourse to the payload data, provided training samples are acquired from which to learn the applicable protocol interaction blocks and extract envelope signal data streams.

Instrumenting Envelope Characteristics

The system and method disclosed herein are related to discriminating the nature of traffic flowing through a computer network. This is a problem that is equivalent to the problem of discriminating between the “sources” of network traffic. Specific considerations and assumptions made as to the groups of traffic being distinguished are discussed herein for the purpose of enabling its application in various settings formed in accordance with certain embodiments of the present invention. Distinguishing types of traffic, for example, sourced by the software application that generates them may be one consideration. Thus, the “source” discriminated by systems formed in accordance with certain embodiments of the present invention may not be tied to a specific web address, but rather to the nature of the operations, including but not limited to web surfing, email exchanges, facebook updates, adware analytics. More specifically a system formed in accordance with certain embodiments of the present invention considers the problem of detecting malicious traffic either originating outside a target network, or inside a target network from a compromised computer, and of distinguishing said traffic from the legitimate web surfing or other applications traffic that may occur. Making such distinctions via the envelope-characteristic-based methods previously described is robust even when the subject traffic is encrypted.

Thus the subject methods become a preferred alternative to systems of prior art such as, so called, “deep packet inspections,” which operations have limited application except in instances where data to be inspected is transmitted in unencrypted clear-text. An additional advantage of this approach is that it avoids invasion of privacy of the users since no data decryption is required. Such methods provide means for constructing a detector of traffic types or groups of traffic types, a detector of traffic that is anomalous relative to typical traffic, and detection and tracking of the reoccurrence of previously observed traffic types.

In certain embodiments, the subject system and method allow for distinguishing between malicious activity and legitimate activity traffic in an encrypted setting. It will be clear from context, however, that any traffic types (application specific, user specific, etc.) may be substituted for “malicious” or “legitimate” categories in practice, and thus the system will learn to make the corresponding distinction between any said traffic types and other typical traffic. It will also be clear from the previous disclosure and from context herein that operation in the encrypted setting is functionally equivalent to operating in the clear-text setting since only exposed envelope packet parameters are exploited—the discussion herein clarifies that, in certain embodiments, the present invention generalizes to cover all classes of encrypted and unencrypted traffic types. Operation on encrypted traffic merely enforces the point disclosed previously, that it is not necessary for a user to examine the data payload of the subject traffic in the course of operations.

In certain embodiments, the first step is to motivate the specific encrypted-traffic problem space. Though once confined to secure transaction activities such as online banking, end-to-end encryption between browsers and servers is now becoming routine even for ordinary browsing and web-based applications. Thus, it becomes challenging for analytics systems to determine the nature of such traffic, other than by relying upon the IP addresses involved. Moreover, this increase in HTTPS traffic over port 443 potentially provides excellent cover for data exfiltration (Ex) activity and command and control (C2) data transacted during network exploitation activity. Since all traffic is encrypted, exploitation activity becomes difficult to distinguish from legitimate HTTPS traffic, and difficult to analyze once detected.

Responsive to this problem, the methods disclosed provide a tool that operates using signature patterns developed exclusively from packet envelope data; the contents of the data payload need never be examined. Certain exemplary embodiments disclosed herein present methods which exploit information such as packet data sizes, channel timings, and packet header data that are exposed features of the internet protocol (IP) stack even when data and commands transferred by the IP stack are encrypted. Details of certain exemplary embodiments formed in accordance with the present invention are elaborated below.

Deep packet inspection and other packet-payload based means of distinguishing traffic types have gained increasing popularity in the cyber security art. Many advanced fire wall analytics systems classify bursts of traffic between a user and a server in terms of its nature; this is done by a combination of examining the IP address of the server and examining the superficial contents of the data exchanged. Thus, for example, such an analytics system may provide a histogram of users' use of services such as, for example, search engines, FACEBOOK, GMAIL, or specific application types such as outlook. However, payload information can be encrypted and IP addresses may be obscured using proxy servers. Thus, it becomes challenging to make these analytic distinctions. The disclosed methods avoid these limitations in detecting and distinguishing any traffic types since they rely on envelope characteristics that cannot be hidden by such actions.

Working from exposed information, signals are constructed in a high-dimensional space representing each observed exchange in each internet conversation. Signature behavior associated with classes of traffic is learned by example. The methods find optimal subspaces within this high-dimensional space in which to distinguish the learned classes of IP traffic from one another. This results in computationally efficient detectors that can quickly screen new traffic for either matches to known signatures or significant deviation from them (i.e., anomalies.)

In an HTTP setting, the systems described can, for example, distinguish between user interaction with different websites and can further differentiate one activity type from another without knowledge of the IP address and without any need for deep packet inspection. The envelope characteristics targeted are properties of the transport (and lower) layers of the IP stack that are not explicitly obscured by HTTPS encryption. Thus, the same envelope signature concepts used to distinguish clear-text IP traffic types can be used to distinguish HTTPS traffic types, without the need for decryption.

A typical setting for such a system is to monitor one or more machines, or to monitor a network or sub-network at an internet gateway point. The subject methods remain highly successful in distinguishing learned IP traffic types even when operating on encrypted port 443 traffic. For example, in certain exemplary embodiments, it is possible to learn and detect signatures for known types of malicious traffic. Moreover, it is possible to abstract signature behavior from a large body of legitimate-traffic samples and to subsequently detect potentially malicious traffic as anomalous relative to those prior examples. Systems formed in accordance with certain embodiments of the present invention are able to differentiate legitimate web activity from malicious traffic without decryption or deep-packet inspections.

There are challenges in generalizing beyond the training sets. This is addressed in certain exemplary embodiments by implementing locally adaptive training in each installation of the technology, thus permitting the detectors to establish what constitutes typical local traffic. A sufficiently broad corpus of training data will provide robust and general characterizations of what constitutes both legitimate and malicious activity patterns that can be applied in various settings for a wide variety of applications.

The subject methods can be deployed both for real-time monitoring of sensitive network gateways, or as an adaptive tool to assist human analysts in review of historical logs. Illegitimate activity to detection by these methods is viable unless and until significant new countermeasure efforts are undertaken by malicious actors. Such counter measures will themselves be detectable since they will require deliberate, anomalous modification of traffic envelope characteristics upon which our detection signals are based.

The methods provide a secure and fast means of monitoring traffic in and out of a network or within a network that provides analytics and security without invasion of the privacy in users' encrypted data.

Application to IP Traffic Irrespective of Encryption

A system and method formed in accordance with certain exemplary embodiments of the present invention work the IP format packet traffic over a typical contemporary intranet or internet system. However, the process is equally applicable to other protocols, with appropriately adapted parameters selected to monitor based upon what is available in packet headers, timing, or other envelope data.

For purposes of clarification, the terms “Packet Chain,” “Action Sequence,” “Sub-Signal,” and “Separation Variable” as used herein take on the following definitions as provided below.

A “Packet Chain” refers to a set of packets chained together by next sequence number and acknowledgment number fields in the packet headers, in an IP conversation between two points—these two points typically being identified by the IP addresses and IP port numbers of the participants. It is well understood that internet traffic exchanges are constructed of groups of such packet chain exchanges, typically resulting from a request or command from one participant yielding a response from the other.

An “Action Sequence” refers to a set of packet chains put together sequentially that reflect a single event of interest—such events may be those as perceived by the user or by a piece of application software or may be arbitrary but self-consistent contacts. For example, a visit to a website by a browser will typically begin with a hand shake, followed by a series of “GET” commands to which the server responds. The initial load of a website is an event that creates a typical action sequence of interest, as would, for example, pushing a button that connects to a new web link from a web page or opens a popup. As described with reference to FIG. 1B, the sequence of exchanges creates an action sequence that is characteristic of the particular operational event of interest. Other examples may include opening a Google calendar session, updating a tweet on a twitter page, initiating a web mail session, reading a web mail letter, updating software, downloading a file, accessing an IP time service, initiating a search, background exchange of adware analytics with a server, etc. An action sequence may also be associated with undesirable activities, such as, among other things, data exfiltration by a Trojan program, an attempt by an external server to install or issue commands to existing malicious software, or attempts by malicious software to contact external controlling server sites, some of which may be deliberately disguised to be superficially similar to the legitimate action sequences.

For practical purposes, action sequences may be limited to a finite length of time. With each action sequence is associated a “signal” determined as disclosed herein, and said signal will be used to determine the classification of the corresponding action sequence. Thus, it is desirable in certain exemplary embodiments of the present invention to limit the action sequence to a span of conversation over which a user wishes to make a unified determination of the class of traffic contained therein.

A “Sub-Signal” is determined on the basis of a packet chain. This sub-signal comprises the set of measurements made upon the packets within the packet chain. In most exemplary embodiments, this includes the vector of sizes associated with each member packet and of Δtime between packets (see discussions of FIG. 1B herein.) In other exemplary embodiments, it includes other information such as the total number of packets in the chain, and further data that may be instrumented from the “envelope” comprising all exposed packet information excluding its actual data payload.

A “Signal” comprises an assembly of joint information from sub-signals for packet chains that make up an action sequence of interest. Thus a signal may include the sequential assembly of the size or Δtime vectors of the sub-signals, preferably (but not necessarily) in a quasi-standardized assembly order. In addition, signals will also preferably include a vector nPackets of the number of packets in each packet chain contributing to the Action Sequence.

It will be apparent those skilled in the art, in light of this disclosure, that other exposed aspects of IP packets may be instrumented similarly to form vector or scalar quantity measurements that comprise signals associated with any give Action Sequence.

One challenge in applying methods formed in accordance with certain embodiments of the present invention to encrypted traffic arises in determining when individual web conversations begin and end in the absence of clear-text cues. Classifier decisions are tied to portions of web conversation (e.g., initial load) referred to as an action sequence. Each action sequence comprises a series of requests and responses between the client browser and the web server. Each of these individual request and response groups comprises a set of packets called a packet-chain is linked by a defined protocol of sequence numbers and acknowledgment codes.

Ordinary HTTP conversations occur in unencrypted clear-text. Thus, for example, each “get” operation is clearly demarked and explicitly labeled in a manner that can be readily parsed by an observer program. It is clear what blocks of packets are linked to which commands, and, more importantly, it is apparent when certain operations occur out of their typical order or, not at all. Browser-server interaction for a given web activity can vary: If a web resource is recently cached by the browser, then it may not request that resource when the page is reloaded. In addition, certain browsers may make multiple, overlapping get requests, to which a server may not respond in the same order on each occasion. Thus any two examples of an action sequence from ostensibly identical web interactions may in fact differ from one another both by omission and transposition of components. This is true even when the underlying content, as perceived by the end-user remains identical. It is further complicated when the content is dynamic, such as when new advertisements are loaded dynamically. Assembly of signals is addressed by examining and matching the labels of the operations such as “get commands” that occur at the initiation of packet chains. Thus, in HTTP conversations, clear-text clues can be used to monitor and compensate for such occurrences.

In HTTPS conversations, users are linked to “blind” parsing since there is not clear text labeling, and therefore users cannot reliably make these adjustments in compensation for the variations. This is by no means show stopping. In an encrypted setting in certain embodiments formed in accordance with the present invention, an order is assumed, while in other exemplary embodiments a method of matched filtering is applied in attempts to best fit the sub-signals with their standardized example components—thus a “nearest neighbor fit” (as detailed below) is applied to each sub-signal within a signal and determine its most likely correspondence. This can be further generalized in certain embodiments by using, for example, a hidden Markov model which is well known in the art of signal modeling to represent the likely transitions from one potentially sub-signal (packet chain type) to another. However, in most instances such processing is too computationally expensive for the high throughput required and thus the alternative embodiments are preferred. In practice, the subject system and method for instrumenting envelope characteristics, specific examples of which are disclosed here, are demonstrated to be remarkably immune to these theoretical limitations.

In applying the methodology to HTTPS data, “blind” parsing may be applied. Operations of the system formed in accordance with certain exemplary embodiments key on conversations linked by IP address and port number, and are further linked by the sequence/acknowledgment codes. Time-gaps within the traffic can be utilized to perform blind parsing of the tokenized units that make up the system's signals. Thus, in an exemplary embodiment, an action signal is considered to persist so long as the IP and port numbers correspond and packet chains continue to be exchanged, until such time as a ½ second time out period elapses. At this juncture, it may be presumed that a new action sequence has begun.

TLS is one protocol known in the art under which HTTPS data is typically encrypted. The initiation of such protocol is marked by exposed handshaking events. This provides a second level of exposed parsing guidance in certain embodiments. Use of these markers provides a different means of segmenting the action sequences. In a preferred embodiment of the subject system, these are combined, and time outs are used to provide reasonable limits to any action sequence.

In accordance with certain aspects of the present invention that action sequence correspondence with any particular underlying, innate segmentation of an IP traffic exchange is NOT critical. The action sequence ultimately results in the tokenized subset of the traffic upon which a decision is made. So long as the parsing mechanism provides consistent results, the training mechanisms provided are transparent to the granularity of segmentation of the data. Thus, for example, if a webpage load is divided arbitrarily into three action sequences by virtue of the parsing mechanism instead of one, as it may be perceived by a human, it is substantially immaterial to the result so long as the parsing mechanism parses the same event(s) the same way at each occurrence.

Another challenge in adapting to HTTPS traffic as compared to HTTP traffic is that the underlying TLS encryption generally employs block rather than streaming ciphers. Thus, packet chains within an action sequence are much more uniform than is typically seen in clear-text HTTP transmissions. This uniformity removes some of the variance exploited in alternative feature extraction approaches. Other exposed envelope characteristics may be suitably exploited to compensate. Some of these are included in our expanded example set below.

The term “Separation Variable” is used herein to describe sets of measurements derived from the signals corresponding to exposed parameters associated with action sequences. Consistent with the disclosed system and method, numerous additional examples are provided. In net result, each parsed action sequence will be reduced in light of its exposed envelope characteristics to a vector of separation variables. This vector of abstracted separation variables is then utilized to make the practical decisions regarding the nature of the source action sequence.

Packets traffic is, for purposes of certain preferred embodiment formed in accordance with the present invention, presumed to be recorded from a gateway point on a network. The given system is therefore able to learn to distinguish classes of traffic transpiring over a gateway between the Internet and the local network. In certain alternate embodiments, the system may be adapted to only instrument within the gateway to learn to distinguish classes of traffic transpiring within the intranet between local computers. Suitable measures known in the art are applied to capture all header data from each packet, together with a time of capture based upon a stable time-keeping system with, preferably, at least millisecond accuracy. Since payload data (encrypted or otherwise) will not be examined, it need not be captured.

FIG. 7A illustrates one configuration of a possible monitor system 78 formed in accordance with certain exemplary embodiments of the present invention which, in a first mode of operation, simply captures packet headers from traffic of interest into a database 78B. The monitor system 78 may capture traffic that moves two and from the gateway via tap point 78C, or, alternatively, may monitor all internal traffic via a bridge connection to the LAN switch 74.

In one exemplary embodiment, the subject system utilizes packets to generate a time and size sub-signal for each packet. The source IP (IPsrc) destination IP (IPdst), source port (prtsrc), destination port (prtdst), next sequence number (nseq), and acknowledgement number (nark) are used while recording the time t, and size s of the packets. For each packet, the packet's source and destination IP and port, as well as its nseq are extracted and followed by the extraction of all packets responding to that packet (i.e., source and destination switch for IP and port, and nack for the response packet is nseq for the source packet):

inds i = find j ⁡ ( ( IP i src = IP j dst ) ( IP i dst = IP j src ) ( prt i src = prt j dst ) ( prt i dst = prt j src ) ( n i seq = n j ack ) )
where indsi indicates the indices of packets belonging to the sub-signal initialized by the ith packet, and findj indicates the function “find all indices j such that . . . is true.” Thereafter, the size and time sub-signal for each packet is computed according to the following equation:
timeSubsigi=[ti,tindsi−[ti,tindsi(1→end−1)]]sizeSubsigi=[si,sindsi]
where timeSubsigi represents the time sub-signal for the ith packet, sizeSubsigi represents the size sub-signal, tindsi represents the time for all indsi packets, tindsi(1→end−1) represents the time for all indsi packets except the last in indsi, and sindsi represents the size for all indsi packets. Thus, the time subsignal indicates the time of the initial packet, followed by the time delta between all sub-signal packets including the initial packet.

In other exemplary embodiments of the present invention, a timeout is applied to truncate sub-signals that have unusually long gaps. In addition, one may require that all packets lie within a finite time window of a reference point. This constrains the overall sub-signal's length as well as the Action Sequence to a designated window in time.

In accordance with an exemplary embodiment of the present invention, the subject system carries out inter-user ranging according to a process such as illustrated in FIG. 9. Generally, the following steps are conducted:

    • 1—Load the set of all packets
    • 2—Loop through each packet
      • a—Determine source/destination IP/Port, next sequence number, and acknowledgement number of current packet
      • b—Find indices of all packets that match IP, Port, and seq/ack number to determine packet-chain groups.
      • c—Store each linked index set as packet chain.
    • 3—Compute sub-signal for each packet chain
      • a—Calculate Δtime and size subsignal
      • b—Record each additional predetermined parameter from the headers or computed counts, e.g., npackets.
    • 4—Store sub signals for each linked packet-chain found.

It is noted that in block 903 and block 904, the criteria for a subsignal as used in a system formed in accordance with certain exemplary embodiments are preferably those indicted above in steps 2a and 2b. Likewise, in block 907 and block 908, the system formed in accordance with certain exemplary embodiments includes computed and header-extracted parameters indicated above in steps 3a and 3b. However, this example is directed toward typical IP traffic as it might occur in the contemporary intent while the precise set of match criteria in any given instance will be adapted to suit the network protocol employed in the monitored system of interest.

Signal Creation:

In accordance with an exemplary embodiment of the present invention, the subject system carries out signal creation which generates the time, size, and packet signals from the set of sub-signals timeSubsigi, and sizeSubsigi. The process of signal creation is carried out, generally, by first calculating the time between sub-signal starts, and identifies those that are below the input threshold (typically set to 0.5 seconds) according to the following equation:
dti=timeSubsigi+1(1)−timeSubsigi(1)inds=findi(dti<ε)
Next, matrices are set up for sub-signals timeSignalsij and sizeSignalsij, so that each element will contain the jth sub-signal for the ith signal. It is noted that the time sub-signals will be stripped of their first element (the time of the initial packet), such that each time sub-signal will be one element less in length than the size sub-signal. The packet signal nPktSignalsi is constructed by simply counting the number of packets in each size sub-signal. Finally, the sub-signals are padded with zeros so that the length of the jth sub-signal remains constant across all signals. This means that for every sub-signal, a signal is constructed, often making overlapping signals. For example, if there are 3 sub-signals before a 1 second pause, 3 signals will be constructed, all ending at the same spot, but each beginning at a different sub-signal.

When constructing signals blindly, only the longest signal for each such endpoint is retained. In systems formed in accordance with certain exemplary embodiments of the present invention, a log of the times of events of interest are employed. Using the logged information, the process begins with the sub-signals identified in the log. In certain alternate embodiments, TLS or other event-start handshaking information is employed. In such embodiments, the process begins with the sub-signals occurring immediately after the handshake event. The systems may be configured to ensure that signals do not overlap and that only the longest such signal for each start point are taken.

In accordance with an exemplary embodiment of the present invention, the subject system carries out pseudo-code using all signals identification according to the following steps once all sub-signals from a single IP address and a timeout E have been entered:

    • 1—Loop through all sub-signals
      • a—Determine all sub-signals whose gap between the 1st element is less than the timeout ε
      • b—Remove initial times from each identified sub-signal
      • c—Store Δtime and size sub-signals as part of a signal
      • d—Calculate nPacket Signal (length of the size sub-signal)
      • e—Store current sub-signal index, and index of the last sub-signal in the signal
    • 2—Zero pad the three sets of signals as needed
    • 3—Store constructed signals and linked sub-signal information

This process is illustrated in general concept in FIG. 10. The optional length normalization step (1006) corresponds to step 2 in the pseudo code described above. The precise form of the operation depends on the context of the particularly intended application, and such process are further elaborated herein, for example in relation to FIG. 3, and specifically feature 30.

In accordance with an exemplary embodiment of the present invention, the subject system carries out pseudo-code using parsing blind signals according to the following steps:

    • 1—Loop through each IP address
      • a—Identify all sub-signals from the current IP address
      • b—Perform All Signals Identification with remaining sub-signals
      • c—Identify each unique end-point sub-signals for the returned set of signals
      • d—Select the signal with the most sub-signals for each unique signal end-point
    • 2—Store signals

This process is broadly illustrated in FIG. 1. The operation All Signals Identification is that described above and in FIG. 10. The goal of this exemplary blind parsing operation is to identify the longest possible signal associated with each possible signal end-point, and thereby provide a well-resolved unique choice among possible signals in the absence of other cues.

In certain alternate embodiments of the present invention, handshaking cues are utilized according to the following steps:

    • 1—Find time and corresponding IP addresses for each SSL/TLS handshake
    • 2—Load Sub-signals
    • 3—Remove sub-signal elements with a timeout gap greater than £
    • 4—Loop through SSL/TLS handshake IP addresses
      • a—Identify all sub-signals from the current IP address
      • b—Perform All Signals Identification with remaining sub-signals
      • c—Find handshake for each sub-signal and identify unique sub-signal associated with the remaining handshake events
      • d—Store only the signals whose 1st sub-signal was identified in c
    • 5—Zero pad the three sets of signals as needed
    • 6—Optional-Strict Handshake: Loop Through each IP
      • a—Determine all signals that don't overlap with other signals
      • b—Remove shorter overlapping signals (keep only the longest one)
    • 7—Store signals

This process is illustrated in FIG. 12 more generally. Step 6 and corresponding blocks 1206-1208 are optional, but preferably enforced in applications where data is abundant in order to maximize the likelihood that the retained signals correspond to unique, repeatable events. In instances where data is less abundant, the omission of these steps leaves a noisier set of signal vectors, yet those vectors still yield tenable results due to the corresponding increase in the size of the sample space. Similarly, block 1209 is optional, and this normalization is described further herein.

Finally, the system accomplishes parsing of signals when a corresponding event log is available. Such a log provides ground-truth verification of specific activities occurring over the channel during the time the data was recorded. For example, and without limitation, the log may be a record of web applications used, or web-pages visited, and specific user initiate operations such as button clicks. Such logs may be available through other surveillance methods employed. They may also be generated in particular where a user or a facility is cooperating in providing training data for a system embodying the present invention. A system thus trained will be later employed to monitor traffic without the benefit of such logs.

In accordance with an exemplary embodiment of the present invention, the subject system carries out pseudo-code using parsing logged signals according to the following steps:

Pseudo-Code: Parsing Logged Signals

    • 1—Load the log file
    • 2—Calculate time for each logged event 3—Load sub-signals
    • 4—Loop through target logged events
      • a—Select sub-signals that begin within the logged window of activity for event
      • b—Sub-select remaining sub-signals that are between the two IPs of interest
      • c—Define signal as the sequence of all remaining selected sub-signals
    • 6—Eliminate from each signal sub-signals with Δt>ε(where Δt is the delta time between the beginning of each sub-signal)
    • 7—Store signals

This process is generally illustrated in FIG. 13. The time of a logged event ti is ordinarily its initiation time, while its duration may be measured or may be fixed at a predetermined window value. The goal of blocks 1302-1304 is to retain only those sub-signals that correspond to the correct time window and IP send/receive addresses as match the logged event type. Block 1305 is an optional noise reduction step which truncates the sequence where excessive delays between packets or packet chains occur. The signal is constructed in block 1306 by concatenating sub-signal components as already further elaborated herein. Each resulting signal after this process is very reliably associated with the logged event. Normalization across repeated events is preferred, and the body of repeated events will typically form high quality training data.

Next, having the underlying envelope data recorded as three signal types, a space called Separation Variables is created based upon abstract measurements computed from this data. A non-exhaustive series of examples of systems formed in accordance with the present invention will first be addressed first in broad concept and subsequently elaborated upon. There are numerous direct and indirect means for numerically monitoring (or “instrumenting”) the properties of the packet envelope data in a group of traffic, other variations of which will become apparent to one skilled in the art given the disclosed approach of instrumenting the envelope rather than the payloads pursuant to the present invention.

One method employed in certain exemplary embodiments formed in accordance with the present invention is to use inner products to determine the degree of match between each signal and a set of prototype signals derived from training data. Thus, the three example signal vectors may be converted into three dimensions of separation variables (timei, sizei, pktsi). In each case, i indexes the set of signals that have been generated in one-to-one correspondence with the set of action sequence collected.

In addition, certain scalar measurements may be conducted directly from the observed data. For example, nPacketChainsi indicates the number of sub-signals in each signal, TotalSizei indicates the total amount of data in the signal, and TotalTimei indicates the total time for the signal excluding certain gaps between sub-signals.

Furthermore, probability-based measures may be constructed based upon the empirical histogram data which is closely related to a Bayesian probability measure derived from experiential frequency counts. Thus, for example, PktChainHistInfi may be defined as the probability that a signal belongs to a group based upon the value of nPacketChainsi; TtlSizeHistInfi as the probability that a signal belongs to a group based upon the value of TotalSizei; PktChainHistNi as the probability that a signal belongs to a training group based upon the value nPacketChainsi; and so forth. Finite N histogram bins may be used to compress the distribution space to a smoother but lower resolution estimate. Thus, TaSizeHistNi indicates the probability that a signal belongs to a training group given the value of TotalSizei; TaTimeHistNi the probability that a signal belongs to a training group is given the value of TotalTimei; and so forth.

Additionally, inner-product similarity measures may be considered on such computed histograms, thus comparing the distributions in their entirety. For example, TimeHistSig1i may be computed as the inner product distance between the normalized histogram of the values in the time signal, with SizeHistSig1i representing the inner product distance between the normalized histogram of the values in the size signal excluding the last bin. One may further choose to weight these distributions by discarding or down-weighting certain bins. Thus, for example, TimeHistSig2i is similar to TimeHistSig1i but excludes the first two bins and the last bin of the histogram, SizeHistSig2i is similar to SizeHistSig1i but additionally excludes the first bin (and continues excluding the last bin), TimeSizeProb1i is the product of TimeHistSig1i and SizeHistSig1i, TimeSizeProb2i is the product of TimeHistSig2i and SizeHistSig2i.

These constructed examples serve to illustrate a range of measures for instrumenting the envelope data of any signal relative to a previously acquired component signature. The separation variables are reduced to scalar values in this exemplary embodiment. The inner products are normalized (such that the sum of each signal is 1). In certain alternate embodiments, the magnitude of the inner products then subtracted from unity to map signals that are closer to each other yield results nearing 0, while more disparate signals are placed near 1. Similarly, the various probabilities may be inverted as 1 minus the histogram probability of the training to again put closer, similar signals near measure 0 and disparate ones near measure 1. These steps are preferred for convenient book keeping but not essential to the disclosed method.

Considering that i indicates the ith signal in the set of all signals, j indicate the jth IP address in the set of all addresses, indsj indicates the set of all i for signals belonging to the jth IP address, mean (Aindsj) indicates the average across all signals in the jth IP address, and hist({Xi}, bin) indicate the histogram of the data {Xi} over the defined set of bins, the following equalities result:

time i = min j ⁢ ( 〈 timeSignals i , mean ⁡ ( timeSignals inds j ) 〉 ) size i = min j ⁢ ( 〈 sizeSignals i , mean ⁡ ( sizeSignals inds j ) 〉 ) pkts i = min j ⁢ ( 〈 nPktSignals i , mean ⁡ ( nPktSignals inds j ) 〉 ) nPacketChains i = length ⁡ ( nPktSignals i ) TotalSize i = sum ⁡ ( sizeSignals i ) TotalTime i = sum ⁡ ( timeSignals i ) C j pInf = hist ⁡ ( nPacketChains inds j , 1 : max i ⁢ ( nPacketChains i ) ) length ⁡ ( inds j ) C j sInf = hist ⁡ ( TotalSize inds j , 1 : max i ⁢ ( TotalSize i ) ) length ⁡ ( inds j ) C j pN = hist ⁡ ( nPacketChains inds j , bin pN ) length ⁡ ( inds j ) bin pN = [ min ⁡ ( nPacketChains i ) ⁢ … ⁢ ⁢ max ⁡ ( nPacketChains i ) ] , in ⁢ ⁢ N ⁢ ⁢ steps . ⁢ C j sN = hist ⁡ ( TotalSize inds j , bin sN ) length ⁡ ( inds j ) bin sN = [ min ⁡ ( TotalSize i ) ⁢ … ⁢ ⁢ max ⁡ ( TotalSize i ) ] , in ⁢ ⁢ N ⁢ ⁢ steps C j tN = hist ⁡ ( TotalTime inds j , bin tN ) length ⁡ ( inds j ) bin tN = [ min ⁡ ( TotalTime i ) , max ⁡ ( TotalTime i ) , in ⁢ ⁢ N ⁢ ⁢ steps ⁢ ⁢ PktChainHistInf i = min j ⁢ ( 1 - C j pInf ⁡ ( nPacketChains i ) ) ⁢ ⁢ TtlSizeHistInf i = min j ⁢ ( 1 - C j sInf ⁡ ( TotalSize i ) ) ⁢ ⁢ PktChainHistN i = min j ⁢ ( 1 - C j pN ⁡ ( bin pN ≡ nPacketChains i ) ) ⁢ ⁢ TtlSizeHistN i = min j ⁢ ( 1 - C j sN ⁡ ( bin sN ≡ TotalSize i ) ) ⁢ ⁢ TtlTimeHistN i = min j ⁢ ( 1 - C j tN ⁡ ( bin tN ≡ TotalTime i ) ) ⁢ ⁢ TimeSigHist i = hist ⁡ ( timeSignals i , bin tSig ) ⁢ ⁢ SizeSigHist i = hist ⁡ ( sizeSignals i , bin sSig ) ⁢ ⁢ ⁢ TimeHistSig ⁢ ⁢ 1 i = min j ⁢ ( 〈 TimeSigHist i , mean ⁡ ( TimeSigHist inds j ) 〉 ) ⁢ ⁢ TimeHistSig ⁢ ⁢ 2 i = min j ⁢ ( 〈 TimeSigHist i 3 : end - 1 , mean ⁡ ( TimeSigHist inds j 3 : end - 1 ) 〉 ) ⁢ ⁢ SizeHistSig ⁢ ⁢ 1 i = min j ⁢ ( 〈 SizeSigHist i 1 : end - 1 , mean ⁡ ( SizeSigHist inds j 1 : end - 1 ) 〉 ) ⁢ ⁢ SizeHistSig ⁢ ⁢ 2 i = min j ⁢ ( 〈 SizeSigHist i 2 : end - 1 , mean ⁡ ( SizeSigHist inds j 2 : end - 1 ) 〉 ) ⁢ ⁢ TimeSizeProb ⁢ ⁢ 1 i = TimeHistSig ⁢ ⁢ 1 i · SizeHistSig ⁢ ⁢ 1 i ⁢ ⁢ TimeSizeProb ⁢ ⁢ 2 i = TimeHistSig ⁢ ⁢ 2 i · SizeHistSig ⁢ ⁢ 2 i
where indsj indicates the indices of the ith IP address of trained signals, and the inner product function

〈 A , B 〉 = 1 - ∑ i ⁢ ( A i A · B i B )
(which indicates one minus the normalized inner product).

The 17 computed dimensions of comparison listed above are intended to provide a general set of examples of various measures that may be computed using the system and method formed in accordance with the present invention. This list is not exhaustive, nor must any particular embodiment use any or all of these particular examples. The first two of these correspond to the explicit examples disclosed in other portions of this Application, and all of these are consistent with the scope and intent of envelope characteristics as accordingly contemplated.

These separation variables define a vector of measurements associated with each action sequence. Learning and subsequent discrimination operations performed are based upon these computed feature dimensions. Thus, if all of the above were computed, the system would operate on a 17 dimensional feature vector corresponding to each action sequence.

Comparison Operations:

Each of the separation variables is intended as a computed measure of similarity between two signals. In practice (and as further discussed with respect to “training” in other sections of this Application) one of these signals will be a predetermined “model” or “prototype” that corresponds to a component signature of one group of interest. Thus, for example, the inner product functions where

〈 A , B 〉 = 1 - ∑ i ⁢ ( A i A · B i B )
are typically implemented with A corresponding to a test signal and B corresponding to an example from a known set. B may be one explicit example from the training set, or it may be a computed from an “average” model signal that is typical of the set.

For instance, timei is shown in the above equations, defined by an inner product computed relative to the vector mean over a set of training signals. This is consistent with the previous disclosed operations formed in accordance with the exemplary embodiment and additional embodiments discussed below.

Nearest Neighbor Operations:

In certain exemplary embodiments, the comparison operation is not with one signal but with a set of signals. This set may correspond, for example, to each example signal in the training set. The example operations are illustrated with min. indicating the minimum over a set of signal comparisons indexed by j. This set may also correspond to one or more pre-computed training examples for each group. For example, there will be one mean signal computed for each known website, thus the min. operation will provide a measure of similarity indexed to the most similar member class in that known set.

Training:

To exploit this data and train a detector to distinguish two groups of data (e.g. in the working example “legitimate” from “malicious”), it must be determined which set will be used to derive the separation variables. This set is designated the “primary training group.” For example, training may be accomplished specifically on the set of malicious examples in order to seek future similarities to this example set. Alternatively, training may be carried out on a very broad set of legitimate examples enabling the future seek signals that are different from the known legitimate examples to detect malicious traffic among these anomalies.

For purposes of subdividing the training classes, the unique IP addresses of this selected training data is identified and a prototype signal is constructed for each IP by the timeSignals, sizeSignals, and nPktSignals for all Action Sequences from the designated IP set within the corpus of training data. This is an optional step taken in the system formed in accordance with an exemplary embodiment of the subject application to enable a nearest neighbor mode based on the assumption that each unique IP address constitutes a natural sub-class of legitimate data. It is not strictly necessary to the invention. Other embodiments may consider the class as a whole or each group (the entirely or the IP subsets) may be divided into further subsets and create a prototype for each subset. Such further division may be accomplished empirically or via some data dimensionality reduction method such as Principal Component Analysis, which results in a set of eigenvectors representative of natural subclasses.

At this point, a set of “component signatures” has been determined for each separation variable, and for each of the IP addresses. The nature of the component signature depends on the nature of the variable in question: Vector component signatures are, in this example, computed as a mean of all such signals across the IP address; probability signatures, in this example, are computed from histogram across signals for the IP address; scalar signatures are computed directly by simply taking the value. These component signatures of the primary training group are stored so that separation variables may be calculated relative thereto in future steps.

This process may be repeated for any number of primary training groups, thus permitting a classifier to make decisions about more than two groups. The non primary training group(s) is used in each case to help establish a threshold of separation between the typical values in the primary group (as established by the component signatures) and non-members of the group. The selected non-primary data groups are referred to as “challenge classes.”

Thereafter, a Nearest Neighbor match operation is performed for each signal in the corpus (primary and non-primary challenge groups), to identify the Separation Variable measured “distance” to the closest Prototypical IP Signal (or IP subset) in the primary training group. This means that each component signature is used to map the separation variables for every Action Sequence to a scalar value for each variable (as described in the previous section). Thus, for a method that employed the example set, the scalars will form a 17 dimensional vector corresponding to each Action Sequence. These dimensions are, in the illustrated embodiment, used pair-wise to find the best SVM separation between the primary and other groups. The separation space parameters (slope of line, threshold point, variable pair) are then stored for each of these optimal separations and the subset of separation spaces (or vote-combined separation spaces, as discussed below) that give the best overall performance is selected.

Related methods known in the art may be applied to make non-linear separations based upon the same derived feature vector without departing from the spirit and scope of the present invention. The above linear separation examples are provided for exemplary purposes only without limitation. Thus, a user may implement a variety of alternative features, including but not limited to similar SVM separations using more than two variables at once, with curved rather than linear separation boundaries, or with non-contiguous collections of boundaries derived from radial basis functions about the training points. One may also consider other known machine classification methods, such as taking further distances, inner products, or other computed metrics in the exemplary 17 dimensional separation variable space discussed herein. The mechanism of deriving similarities by taking measurements of packet envelope information rather than payload data to train such systems remains a fundamental distinguishing aspect from prior art.

FIG. 6C illustrates a linear partition SVM generated in accordance with certain embodiments of the present invention. In FIG. 6C, two derived separation variables are provided for instance, although only two derived separation variables are employed for ease of visualization and because two dimensions generally provides significant computational speed advantages over higher dimensional operations. Nevertheless, it should be appreciated that higher dimensional partitions may be utilizes without departing from the scope of the present invention. Again, SVM, is used here to refer to a range of known training-example-based classification method that rely on finding an division line, plane, or hyper-plane that substantially segregates the members of two or more classes of data in a projected space with dimensions formed from two or more separation variables.

The two separation variables 61 and 62, indicated as X15 and X13, may comprise any of the above listed measures or any similar envelope derived measure. In this embodiment, separation variables 61 and 62 have been scaled to range from 0 to 1. The chart shows two primary clouds of point, the light crosses 63, which correspond to data from the legitimate traffic samples in the training data, and the dark crosses 64, which correspond to the malicious traffic samples in the training data. Each point thus corresponds to a representation in this two dimensional subspace of the envelope characteristics derived from one action sequence within one of the two training groups. Therefore, as discussed above, each of the action sequences (as was parsed by the given parsing system) is judged in its entirety as to its class membership. The SVM system is configured through suitable means known in the art to find a separation partition 65 which optimally divides the two groups within this particular subspace. In this instance, one primary and one challenge class are utilized. This optimization may be relative to various criteria in different embodiments, including minimizing the number or proportion of incorrect classifications in each training group.

As FIG. 6 illustrates, the bulk of the malicious traffic is represented by a cloud of crosses 64 and the bulk of the legitimate traffic is represented by a second cloud of crosses 63, where each is substantially segregated from the other by the division boundary 65. However, separation is not perfect; in FIG. 6 each incorrect point is circled. Thus, for example, the malicious traffic point 66A and the legitimate traffic point 66B are on the wrong side of the boundary line 65. The count of incorrect classifications provides a measure of expected accuracy of this classifier.

Any pair of separation variables Xa and Xb may be similarly projected into a decision plane and an optimal boundary determined relative to this training data. Triples, quads, and other such dimensions of separation variables may be employed with separation boundaries determined with planes and hyper surfaces in other embodiments, with new points projected into corresponding higher dimensional spaces.

Non-linear separations may be likewise employed. For comparison, FIG. 6D illustrates one example of a nonlinear separation space determined via radial basis functions. In this instance, rather than attempting to segment the points with a straight line, the system attempts to find smooth boundaries that maintain a constant weighted distance from the maximum number of points in each class. The resulting division in this example is neither linear, nor even connected. The light malicious points are separated from the dark legitimate points by three distinct, noncontiguous bounded regions. The lighter interior or each of these corresponds to a “malicious” classification, while the darker exterior field corresponds to a “legitimate” classification region.

An illustrative training procedure of an exemplary system formed in accordance with certain embodiments of the present invention performs the following steps:

1—Capture Packets

2—Find Packet Chains

3—Compute Sub-Signals for each packet chain

4—Parse Action Sequences according to selected method

5—Generate Signals for each Action Sequence

6—Segment signals by classes of interest

7—For each primary class

    • 7a—For each subclass determined within primary class
    • 7aa—Compute component signatures according to envelope data type.
      8—Select or synthesize challenge class data
      9—For each signal in primary and challenge classes
      9a—Compute separation variables relative to component signatures
      10—For each primary class
      10a—Perform SVM separation optimizations for each primary and challenge class pair, for each pair of separation variables selected.
    • 10b—Select optimal classification subspace (s) based on performance in training data.
      11—Store classifier data for each primary class classifier set including: component signatures, classification subspace separation variables and partition data.

This process is illustrated in general concept in the flowchart of FIG. 14. Preferably, in block 1401 only the packet headers and timing information are retained since these contain all actionable envelope data. In block 1402, packet-chains are identified and corresponding sub-signals are formed such as described in preceding paragraphs. Likewise, in block 1403 the sub-signals are concatenated according to means discussed to form signals for each action sequence. For purposes of building a useful classifier, these training signal examples are then divided according to the classes of interest for future detection and discrimination. For each of these classes (and, as indicated in the pseudo code above, in certain embodiments sub-classes thereof) a representative signature is computed. This representative signature is typically generated for each signal component type. Thus, if signals exist for delta-time, size, and nPacket vectors, then a component signature will be computed for each vector. As previously discussed, numerous other measurement parameters may be included to enable computation of, for example, 20 or 30 different component signatures.

Each component signature forms the respective basis for comparing each primary class with each challenge class. As indicated in block 1406 and in the pseudo code above, challenge data may be generated or selected from other recorded signals that are of different classes than the primary classes of interests. These challenge classes provide points of comparison. Thus, for each signal in the primary class and for each signal in each challenge class (block 1407), a separation variable by which we may measure the difference between the “typical” signature for the class and challenge class are generated. In the event that sub-classes are employed, the “typical” signature is replaced by a set of “typical” signatures for each subclass and a nearest neighbor test employed.

Each signal in the primary class and each signal in each challenge class is thus reduced to a number, for each separation component, by which we may test its similarity to the typical signal for that separation component in a particular primary class. In accordance with block 1408, optimization is conducted by finding the best separation boundary between each such class and each corresponding challenge class, taken pair-wise, for each of the components of separation. This is typically accomplished with a variation on linear or non-linear SVM. Preferably, optimization is achieved pair-wise in as low a dimensional space as feasible, while maintaining given accuracy constraints, for speed of operation. Thereafter, as illustrated in block 1409, an optimal classification sub-space is selected for each pair-wise comparison between each class of interest and each challenge class, and the information is stored for subsequent retrieval and use.

Classify Data Based on Existing Separations:

In order to classify novel data, training results are used. First, data that was used to derive the separation variables in the training stage is loaded. Next, the set of component signatures for the trained data is recalculated or loaded as necessary, and a Nearest Neighbor operation for the new data is performed. Subsequently, the separation spaces defined in the previous section are loaded to determine in which group each signal lies. This determination is made by plotting each corresponding separation value vector for the new data into the existing spaces and examining on which side of the separation lines the new data falls.

In FIG. 6C, a test point 67 is illustrated, which corresponds to the separation variable values obtained for a novel action sequence in the traffic. It is plotted into the space and, by comparison with the division boundary 65, determined to fall on the “legitimate” side of the classification sub-space. Therefore, this particular example classifier would classify the new traffic segment as “legitimate.” This reflects abstractly that the measured envelope characteristics of the novel action sequence are more consistent with the legitimate set of examples than with the malicious examples.

As discussed with regards to training, other machine learning methods known in the art may be employed, in which case the method employed for training is also used in the classification step accordingly. Non linear boundaries, higher dimensions, and other parametric variations may be applied, depending on the particular requirements of the intended application.

An alternative to higher dimensional decision spaces is to use a voting scheme based upon groups of two dimensional separation spaces. Thereby, exhaustive testing enables a user to find a well performing pair wise space Xa and Xb and a second space, Xc and Xd, Xe and Xf, etc. A test point such as 67 would be generated in each pair wise separation space, and classified according to each spaces learned separation division. The results would be tallied in a vote, with the class of the novel action sequence determined by the largest number of votes. This system permits results from independent subspaces to reinforce one-another, often generating higher accuracy than any one subspace, but with less computational demands than a higher dimensional system. Such a system may also be employed to generate votes among any number of classes larger than two, thus classifying a novel action sequence to its best fit among many possible classes.

Such a system may be employed, for example, to divide traffic automatically by website of origin. FIG. 8 illustrates a confusion matrix showing high accuracy at separating websites by their traffic source even when the data is encrypted and IP address are hidden from the system. Multiple classes may also be used to distinguish applications types for traffic, and other parameters of interest for monitoring and use-case analytics as discussed in the introduction. The commercial website names have been changed in the figure so as not to infringe their owners' rights; however, the overall accuracy is nearly 90% and traffic from each of these websites was recorded from encrypted HTTPS sessions. As exhibited in FIG. 8, such voting may create a “null” space which is defined as those signals for which there is a tie in the number of votes thus rendering the decision indeterminate. Tie breaking procedures may be employed in certain exemplary embodiments.

Operational Classification Procedure

An illustrative operational classification procedure of a system formed in accordance with certain exemplary embodiments of the present invention includes the following steps:

1—Capture Packets

2—Find Packet Chains

3—Compute Sub-Signals for each packet chain

4—Parse Action Sequence(s) according to selected method

5—Generate Signal for at least one Action Sequence

6—Load classifier data.

7—For each primary class

    • 7a—Compute separation variables for Signal(s) relative to primary class/sub-class component signatures.
    • 7b—Compute nearest-neighbor separation variables for Signal(s) relative to primary class.
      8—For each pre-determined separation subspace.
    • 8a—Project separation variables into SVM separation space.
    • 8b—Determine classification of action sequence relative to SVM division boundary.
      9—(Optionally) combine subspaces by voting to determine winning primary class.
      10—Report classification for novel Action Sequence(s)

This process is illustrated in FIG. 15 more generally. Preferably only headers and occurrence times need be captured (block 1501). The parsing, as indicated by blocks 1502 and 1503, occurs as before, consistent with the corresponding training data. Only one action sequence or a set may be thus parsed and further processed at any given time.

In block 1504, computed separation variables relative to primary signatures are established for each component type, for each class, and for each sub-class if employed (e.g., 7-7a-7b in the steps above).

In accordance with block 1505, a user 1505 projects into each of the separation spaces previously selected during training as optimal for classifying known examples. The data from the new Action Sequence will be thus classified in each space by virtue of where it falls relative to the corresponding class separation boundary. In the event that many pair-wise comparisons are employed, voting is preferably employed to determine the winning class from amongst the choices. This is reported as the output of the process.

Applying various features disclosed herein to IP traffic separation begins with defining traffic types of interest. In accordance with exemplary embodiments discussed herein, the distinction between “malicious” versus “legitimate traffic” is used; however, any traffic type may be targeted. A collection of training data is then established, comprising a examples of traffic known to be of the class of interest. This may be done by ensuring only appropriate traffic examples occur during a given period, by creating example traffic that goes to tagged IP addresses, or by logging events of interest and coordinating these logs with a time record of recorded traffic.

The packets header data is captured in accordance with the criteria described above, and packet chains are collected from the captured data and parsed by any suitably accommodated mechanism to effectively group them into action sequences. For each packet chain a sub signal is created, and for each action sequence the sub-signal data is aggregated into signal data.

At least one primary training group of data is established, and from this component signatures are determined for use in computing a separation variable vector for each action sequence within the training group.

At this point, the traffic has been parsed, packet chains have been mapped to sub-signal data, and packet chains have been grouped into action sequences and the corresponding sub-signal data aggregated into signals. Thereafter, predetermined separation variables are computed from the aggregate signal data, thus mapping each action sequence to a well-defined multi-dimensional feature vector upon which further operations occur.

Envelope Instrumentation Process

1—Separate training traffic into packet chains

2—Compute sub-signals from exposed envelope information of packets within each packet chain

3—Parse action sequences corresponding to consistently tokenized traffic events

4—Aggregate sub-signals into signals based upon action sequence parsing

See FIG. 16 for flow chart, the details of operation in each process block for exemplary embodiments having been further elaborated herein.

Training Process

1—Record training traffic samples

2—Perform Envelope Instrumentation Process on recorded traffic

3—Separate signals into groups based on predetermined traffic classes of interest

4—Select a primary training group

    • Compute component signatures for primary training group
    • Compute separation variables for all signals in all groups
    • For each separation sub-space
      • find optimal separation division between primary and other group
    • Find optimal separation sub-space(s)
    • (Optional) find optimal combined set of sub-spaces according to voting scheme.
    • Store component signatures and SVM separation space details
      5—Repeat for any additional primary training groups

See FIG. 17 for flow chart the details of operation in each process block for exemplary embodiments having been further elaborated herein.

Operational Classification Process

1—Record new traffic sample

2—Perform Envelope Instrumentation Process on recorded traffic

3—Select a set of primary training components signatures

4—Compute separation variables for new signal based on primary training components signatures

5—Project computed separation variables into previously determined SVM separation space

6—Report classification for the new traffic signal based on which side of SVM division it falls.

7—(optionally) Aggregate classification results via voting to determine final classification results

See flowchart FIG. 18, the details of operation in each process block for exemplary embodiments having been further elaborated herein.

Example System Embodiments for Certain Applications:

FIG. 7A illustrates one potential application for a system formed in accordance with exemplary embodiments of the present invention. A local area network (LAN) comprising a set of network connected computers (and potentially other peripherals) 75-77 is linked via a switch 74 and is tied via a gateway 73 to the internet 70. Remote sites 71 and 72 are represented as boxes on the internet cloud. Thus, for example, computer system 75 may communicate via a browser or other network application with a remote server 71, computer 76 with the same server 71 or different server 72, as demanded by users and/or their running applications.

A system monitor employing the subject methods 78 is connected to a tap point 78C, thus it has access to all inbound and outbound traffic to and from the web. Alternatively, this system may be attached to the LAN switch 74 in a bridging mode so that it can monitor all internal traffic. Regardless of connection mode, the monitoring system may employ various means to filter and collect packet traffic as known in the art. Traffic is recorded to a database 78B for future inspection via a review system 79 shown FIG. 7B. In alternative embodiments, the traffic is inspected in real time at the Monitor system 78. Thus, the various embodiments described herein may be employed either by real time monitoring or via retrospective analysis of recorded traffic, or both.

Consider a scenario in which the Monitor system is configured via certain embodiments of the subject system and method to detect and record the type of application data moving through the gateway 73. Thus packets are recorded at 78C, the system instruments these via suitable processes methods above to produce envelope characteristic signals. In a training mode, the system is exposed to typical traffic to each application type of interest, for example, web mail, browsing, and a proprietary company service Z. Examples of each of these are logged in the database 78B, with packet chains and correspondingly parsed action sequences established. Classification spaces are then generated in accordance with certain aspects of the present invention, and saved. In a subsequent detection mode, the system subsequently takes each new observed action sequence that crosses 78C and produces a set of separation feature measures in accordance with those determined during training to be preferable in making determinations. Each instance of traffic of each type to the database 78B is then recorded to provide a log of user activity, statistics or use, for example, without decrypting data or invading users' privacy more than is necessary to obtain these statistics.

Alternatively, in accordance with FIG. 7B, these traffic events may simply be recorded and then analyzed later using the system in a retrospective review mode. A variation on this scenario is contemplated in other embodiments described herein, wherein monitoring is carried out as to which websites are accessed by users on a LAN via their envelope characteristics and reporting when an apparent activity is not consistent with the website IP address accessed. Such anomalous events may be discovered in retrospective analysis. However, in other embodiment such events may be tested for and discovered in real-time. In that case, the system 78 would log the event in 78b for later inspection.

In some instances, it may be desirable for the monitor system 78 to be configured to watch for malicious internet traffic such as for contact between a Trojan virus software on one computer 75 and a malicious external server 72. This traffic may be encrypted and embedded in other HTTPS traffic over port 443, and thus be difficult or impossible for firewall or packet inspection analytics system of prior art to detect. To configure the system for this scenario, the system is first trained by exposing it to a broad selection of typical traffic over the gateway 73. Next, the data is divided into reasonable subsets, such as by IP address or known traffic types, and component signatures are developed.

Anomalous activity (either real or simulated) is then used as counter examples to this typical training data in order to generate one or more optimal separation space classifiers. These anomalies may be synthesized using probability distributions where examples of malicious traffic are not readily available or predictable. Thus, there is no requirement in the given embodiment to use examples of two classes of interest in an anomaly detection scenario. The only requirements are that there be examples of one class of interest and that division spaces are optimized such that a predetermined significant portion of the “typical” data falls on one side of the division barrier. Preferably, real malicious training examples will also be included to aid optimization in detecting anomalies that might be closely similar to the legitimate examples in certain dimensions of envelope measurements. The system thus obtains a set of optimized classifiers based on the legitimate traffic sample set, and the anomalous traffic or simulations available.

The monitor system is then activated in an active testing mode. Each new traffic event is parsed into action sequences and each action sequence is reduced, in accordance with certain aspects of the present invention, to a representative vector of separation variables based upon the action sequences' aggregate envelope characteristics. Thus, in accordance with the classification mode described above, each action sequence is determined to be consistent with known legitimate traffic, or it is determined to be anomalous.

A real-time mode embodiment of the monitoring system 78 may act upon this information immediately by any of several practical means. In an exemplary embodiment formed in accordance with the present invention, it may simply log events to the database 78B for later review. Alternatively, the system may be configured to sound an alarm via an alert subsystem 78A to call human attention to the event. Furthermore, direct action may be taken consistent with “firewalling” the anomalous traffic by interrupting it at the gateway link point 78C. Still further, the system may actively track this event and all follow-on events between the suspected parties to provide a chain of relevant details for subsequent in depth study.

Once examples of specific malicious activity have been obtained by any suitably accommodated means, including but not limited to anomaly detection or human analysis, they may be used to establish tracking and detection of future similar events. Thus, the malicious examples may be employed as the primary training set from which component signatures are generated and classification spaces established in accordance with the invention using separation variables calculated relative to these signatures. In active testing mode, the monitoring system would therefore be enabled to directly detect examples of specific known types of malicious activity traffic that transpire across the network. Such a system can adopt a host of actions as described in preceding paragraphs to directly address such events or by logging them for future review.

In certain embodiments, a system formed in accordance with the present invention may be configured such that each of the detector modes discussed herein operate simultaneously. For instance, the system may simultaneously detect classes of ordinary web activity, track ordinary and unusual website traffic activity, detect traffic which is anomalous as compared to previous examples of known legitimate traffic, and detect specific known malicious activity patterns by their signatures. This combined set of detection systems may be preferred in certain applications where it is desirable to have intelligent interaction between the detectors in making final determinations about any given traffic, and to provide a multitude of analytics services to an end user without the necessity for deep packet inspection or decryption of secure data payloads.

Where the monitoring system 78 examines traffic crossing the LAN switch 74, similar analytics may be performed on the internal network traffic. Exchanges are recorded, for example, between computers, and between computers and internal servers and peripherals, to provide records of the activity that do not require decryption of payload data. Similarly, an internal LAN system may be monitored for malicious activities, including for example: the spread or attempted spread of a Trojan virus or worm from one machine to another, unauthorized activity types, or attempts to gain access to network connect systems by unauthorized internal parties.

The system and method disclosed herein will have broad application apparent to those skilled in the art once they have understood the present disclosure. Upon reviewing the novel combinations of elements disclosed in the specification and figures and the teachings herein, it will be clear to those skilled in the art that there are many ways in which the subject system and method may be implemented and applied. The description herein relates to the preferred modes and example embodiments of the invention.

The descriptions herein are intended to illustrate possible implementations of the present invention and are not restrictive. Preferably, the disclosed method steps and system units are programmably implemented in computer based systems known in the art having one or more suitable processors, memory/storage, user interface, and other components or accessories required by the particular application intended. Suitable variations, additional features, and functions within the skill of the art are contemplated, including those due to advances in operational technology. Various modifications other than those mentioned herein may be resorted to without departing from the spirit or scope of the invention. Variations, modifications and alternatives will become apparent to the skilled artisan upon review of this description.

That is, although this invention has been described in connection with specific forms and embodiments thereof, it will be appreciated that various modifications other than those discussed above may be resorted to without departing from the spirit or scope of the invention. For example, equivalent elements may be substituted for those specifically shown and described, certain features may be used independently of other features, and in certain cases, particular combinations of method steps may be reversed or interposed, all without departing from the spirit or scope of the invention as defined in the appended claims.

Claims

1. A system for distinguishing between a plurality of remote sites accessed through a network interconnection by a local site based upon envelope characteristics of communication signals transmitted in packets therebetween, comprising:

a capture processing portion executing to time-capture a plurality of segments of the communications signals transmitted during an interconnected session established between one of the remote sites and the local site;
a parsing processing portion coupled to said capture processing portion to selectively generate for each of the interconnected remote sites at least one envelope signal indicative of at least one resource allocation response thereof during the interconnection session, each said envelope signal being separately defined by values of at least one predetermined envelope parameter acquired from the time-captured segments, said parsing processing portion being executable to: separate the time-captured segments into corresponding chain segments within at least one action sequence based on at least one predetermined packet header parameter and an inter-packet time gap parameter of the communication signals, form for each remote site a plurality of action sequences each defined by a corresponding chain of packet segments, successive action sequences being delineated based on the inter-packet time gap parameter of the communication signals, form an envelope sub-signal for each of the action sequences according to the predetermined envelope parameter values, and concatenate a plurality of said envelope sub-signals to form said envelope signal; and,
a signature processing portion coupled to receive said envelope signal from said parsing processing portion for each interconnected remote site to be identified, said signature processing portion executing responsive to said envelope signal to generate a characteristic signature for uniquely identifying the interconnected remote site, newly-captured communication signal segments being thereby classified in remote site origination based on said characteristic signatures of identified remote sites.

2. A method for distinguishing between a plurality of remote sites accessed through a network interconnection by a local site based upon envelope characteristics of communication signals transmitted in packets therebetween, comprising:

executing a processor for time-capturing a plurality of segments of the communications signals transmitted during an interconnected session established between one of the remote sites and the local site;
executing a processor for parsing the time-captured segments to selectively generate for each of the interconnected remote sites at least one envelope signal indicative of at least one resource allocation response thereof during the interconnection session, each said envelope signal being separately defined by values of at least one predetermined envelope parameter acquired from the time-captured segments, said parsing including: separating the time-captured segments into corresponding chain segments within at least one action sequence based on at least one predetermined packet header parameter and an inter-packet time gap parameter of the communication signals, forming for each remote site a plurality of action sequences each defined by a corresponding chain of packet segments, successive action sequences being delineated based on the inter-packet time gap parameter of the communication signals, forming an envelope sub-signal for each action sequence according to the predetermined envelope parameter values, and concatenating a plurality of said envelope sub-signals to form said envelope signal; and,
executing a processor to establish a characteristic signature responsive to said envelope signal generated for each interconnected remote site to be identified, said characteristic signature uniquely identifying the interconnected remote site, whereby newly-captured communication signal segments are classified in remote site origination based on said characteristic signatures of identified remote sites.

3. A method for distinguishing between a plurality of remote sites accessed through a network interconnection by a reference site based upon envelope characteristics of communication signals transmitted in packets therebetween, comprising:

executing a processor to carry out envelope instrumentation on the communication signal packets, said envelope instrumentation including: time-capturing a plurality of segments of the communications signals transmitted during an interconnected session established between one of the remote sites and the reference site; blind parsing the time-captured segments to selectively generate for each of the interconnected remote sites at least one envelope signal indicative of at least one resource allocation response thereof during the interconnection session, each said envelope signal being separately defined as a time-varying waveform by values of one predetermined envelope parameter acquired from the time-captured segments, wherein said blind parsing includes: separating the time-captured segments into corresponding chain segments within at least one action sequence based on at least one predetermined packet header parameter and an inter-packet time gap parameter of the communication signals; forming for each remote site a plurality of action sequences each defined by a corresponding packet chain, successive action sequences being delineated based on the inter-packet time gap parameter of the communication signals; forming an envelope sub-signal for each of the action sequences according to said predetermined envelope parameter values; and, concatenating said envelope sub-signals to form said envelope signal, and,
executing a processor to establish a characteristic signature responsive to said envelope signal generated for each interconnected remote site to be identified, said characteristic signature uniquely identifying the interconnected remote site, whereby newly-captured communication signal segments are classified in remote site origination based on said characteristic signatures of identified remote sites in payload data-independent manner.

4. The system as recited in claim 1, wherein the communication signals are transmitted in a Transmission Control Protocol/Internet Protocol (TCP/IP) format, and said parsing processing portion links the chain segments within at least one action sequence based on at least one of: an IP address parameter, a port number parameter, a sequence number parameter, and an ack number parameter acquired from the packet header of the communication signals.

5. The system as recited in claim 1, wherein the communication signals are transmitted in encrypted form, and said parsing processing portion executes blind parsing to selectively generate said envelope signal indicative of at least one resource allocation response occurring during an HTTPS secured interconnection session.

6. The system as recited in claim 5, wherein said blind parsing processing portion executes to delineate at least one of the action sequences from another based at least in part on acquisition of a predetermined event selected from the group consisting of: a communication protocol handshake event, a logged event, and a timeout event.

7. The system as recited in claim 1, wherein said capture processing portion executes to record packets of communication signal data from at least one network gateway point, the packets being recorded exclusive of payload data.

8. The system as recited in claim 1, wherein:

the remote sites are each websites on the world wide web, and the local site corresponds to a client connected to the world wide web configured to communicate with the websites in accordance with an HTTPS secured protocol; and,
each resource allocation response of a website includes accessing at least one resource thereof by Universal Resource Identifier (URI).

9. The system as recited in claim 8, wherein:

each of the time-captured segments represents a packet of communications data transmitted through the world wide web according to the Transmission Control Protocol/Internet Protocol (TCP/IP);
each of the time captured-segments includes time and packet length information for the communications data transmitted thereby; and,
said predetermined envelope parameter is defined by at least one of the time information and packet length information.

10. The system as recited in claim 1, further comprising a classifier processing portion coupled to said parsing processing portion, wherein said parsing processing portion generates said envelope signal for newly-captured communication signal segments from an interconnection session established between an unidentified remote site and the local site, said classifier processing portion classifying the newly-captured communication signal segments in remote site origination responsive to comparison of said envelope signal thereof with said characteristic signatures of identified remote sites.

11. The method as recited in claim 3, wherein:

the remote sites are each websites on the world wide web, and the reference site is a client connected to the world wide web configured to communicate with the websites in accordance with an HTTPS secured protocol;
each resource allocation response of a website includes accessing at least one resource thereof by Universal Resource Identifier (URI); and,
each of the time-captured segments represents a packet of communications data transmitted through the world wide web according to the Transmission Control Protocol/Internet Protocol (TCP/IP).

12. The method as recited in claim 11, wherein:

each of the time captured-segments includes time and packet length information for the communications data transmitted thereby; and,
said first predetermined envelope parameter is defined by at least one of the time information and packet length information.

13. The method as recited in claim 11, further comprising executing classifier processing to:

apply at least one predetermined comparison metric to compare each said envelope signal of newly-captured communication signal segments with at least one of said characteristic signatures of identified remote sites, said classifier processing portion generating for each comparison a comparison value corresponding to each said predetermined comparison metric; and,
apply a predetermined decision process upon said comparison values to classify each newly-captured communication signal segment in remote site origination, said predetermined decision process being selected from the group consisting of: a linear support vector machine (SVM) process, a quantitative thresholding process, and a voting tree process.

14. The method as recited in claim 11, wherein said blind parsing executes to link the chain segments within at least one action sequence based on at least one of: an IP address parameter, a port number parameter, a sequence number parameter, and an ack number parameter acquired from the packet header of the communication signals.

15. The method as recited in claim 11, wherein:

said blind parsing processing executes to delineate at least one of the action sequences from another based at least in part on acquisition of a predetermined event selected from the group consisting of: a communication protocol handshake event, a logged event, and a timeout event; and,
said time-capturing executes to record packets of communication signal data from at least one network gateway point, the packets being recorded exclusive of payload data.

16. The method as recited in claim 2, wherein:

said parsing includes: forming for each remote site a plurality of action sequences each defined by a corresponding chain of packet segments, successive action sequences being delineated based on the inter-packet time gap parameter of the communication signals; and, forming an envelope sub-signal for each of the action sequences according to the predetermined envelope parameter values; and,
said envelope signal is formed by concatenating a plurality of said envelope sub-signals.

17. The method as recited in claim 16, wherein:

the communication signals are transmitted in a Transmission Control Protocol/Internet Protocol (TCP/IP) format, and said parsing executes a blind parsing process to link the chain segments within at least one action sequence based on at least one of: an IP address parameter, a port number parameter, a sequence number parameter, and an ack number parameter acquired from the packet header of the communication signals; and,
the communication signals are transmitted in encrypted form, and said parsing processing portion executing to selectively generate said envelope signal indicative of at least one resource allocation response occurring during an HTTPS secured interconnection session.

18. The system as recited in claim 17, wherein:

said parsing executes to delineate at least one of the action sequences from another based at least in part on acquisition of a predetermined event selected from the group consisting of: a communication protocol handshake event, a logged event, and a timeout event;
said time-capturing executes to record packets of communication signal data from at least one network gateway point, the packets being recorded exclusive of payload data;
each of the time-captured segments includes time and packet length information for the communications data transmitted thereby; and,
said predetermined envelope parameter is defined by at least one of the time information and packet length information.

19. The method as recited in claim 16, wherein:

said blind parsing process generates said envelope signal for newly-captured communication signal segments from an interconnection session established between an unidentified remote site and the local site; and,
the newly-captured communication signal segments are classified in remote site origination responsive to comparison of said envelope signal thereof with said characteristic signatures of identified remote sites.
Referenced Cited
U.S. Patent Documents
5400261 March 21, 1995 Reynolds
5436653 July 25, 1995 Ellis et al.
5437050 July 25, 1995 Lamb et al.
5787253 July 28, 1998 McCreery
6147976 November 14, 2000 Shand
7486799 February 3, 2009 Rhoads
7620807 November 17, 2009 Spatscheck et al.
7644150 January 5, 2010 Nucci
7720013 May 18, 2010 Kelliher
7944822 May 17, 2011 Nucci
7986913 July 26, 2011 Wang
8135091 March 13, 2012 Alexander et al.
8180916 May 15, 2012 Nucci
8331234 December 11, 2012 Newton
8516586 August 20, 2013 Jensen
8577817 November 5, 2013 Keralapura
8676729 March 18, 2014 Keralapura
8788650 July 22, 2014 Xie
8843627 September 23, 2014 Baldi
8964548 February 24, 2015 Keralapura
20030108042 June 12, 2003 Skillicorn
20050044406 February 24, 2005 Stute
20050086520 April 21, 2005 Dharmapurikar
20050188423 August 25, 2005 Motsinger
20050281291 December 22, 2005 Stolfo
20060107321 May 18, 2006 Tzadikario
20060143710 June 29, 2006 Desai
20070011317 January 11, 2007 Brandyburg
20070143847 June 21, 2007 Kraemer
20070171827 July 26, 2007 Scott
20070214504 September 13, 2007 Milani Comparetti
20080028468 January 31, 2008 Yi
20080123545 May 29, 2008 Watanabe
20090031420 January 29, 2009 Lloyd
20090089869 April 2, 2009 Varghese
20090099988 April 16, 2009 Stokes
20090129288 May 21, 2009 Hernacki
20090161544 June 25, 2009 Kelly
20100071063 March 18, 2010 Wang
20100077482 March 25, 2010 Adams
20100124182 May 20, 2010 Han
20100205297 August 12, 2010 Sarathy
20100205665 August 12, 2010 Komili et al.
20100238835 September 23, 2010 Lundgren et al.
20100284300 November 11, 2010 Deshpande
20110040706 February 17, 2011 Sen
20110149793 June 23, 2011 Kim
20110182290 July 28, 2011 Perkins
20110314269 December 22, 2011 Stavrou
20120047096 February 23, 2012 Duffield
20120113857 May 10, 2012 Narayanaswamy
20120190380 July 26, 2012 Dupray et al.
20120216265 August 23, 2012 Mansour et al.
20120221497 August 30, 2012 Goyal et al.
20120224617 September 6, 2012 Feher
20120240185 September 20, 2012 Kapoor
20120317306 December 13, 2012 Radinsky
20130070622 March 21, 2013 Degioanni
20130083203 April 4, 2013 Barrett
20130107715 May 2, 2013 Szabo
20130114612 May 9, 2013 Singh
20130194949 August 1, 2013 Ruddick
20130254884 September 26, 2013 Dalcher
20140150102 May 29, 2014 Wang
20150172312 June 18, 2015 Wang
Other references
  • Fang Yu, et al., “Gigabit Rate Packet Pattern Matching Using TCAM,” Computer Science Division (EECS),pp. 1-12.
Patent History
Patent number: 9813310
Type: Grant
Filed: Mar 18, 2014
Date of Patent: Nov 7, 2017
Assignee: Reality Analytics, Inc. (New York, NY)
Inventor: Jeffrey Mark Sieracki (Silver Spring, MD)
Primary Examiner: Philip Chea
Assistant Examiner: Wing Ma
Application Number: 14/218,435
Classifications
Current U.S. Class: Pathfinding Or Routing (370/351)
International Classification: G06F 15/173 (20060101); H04L 12/26 (20060101);