Let us take the pharmaceuticals industry as an example. There is a list of highly specific industry news sites publishing information about new drug releases and their trials. Alongside this information a very different, but still pharma related, type of articles is published - reports about management changes, company mergers and acquisitions. Both types of articles are very relevant for anyone following the state of pharma industry. Yet keeping track of such diverse information is difficult.

To address issues like this TokenMill builds media intelligence systems capable of automatically classify articles by its type. In this case two classes of business and drug production events are introduced. For the drug production related class of articles we then start a text analysis engine to identify: drug ingredients and their names, disease names and symptoms or drug testing stages. For the business articles we detect company names, locations and person names. Once all this information is extracted, business insights about pharma products and corporate events can be generated.

Let us take a government organization that works with regulation documents published by different legislatures: EU and local level as well as various internal procedural documents. The language of those documents often is very different from the one used by individuals formulating search requests. Legislative documents use highly specific vocabulary which might not match the one used by the users working in different departments. Thus we need a mechanism translating whatever is entered to the search box into an information request.  Then the documents matching a user's query have to be ranked according to their importance. Legal documents might not have any references to each other in order to help rank them by importance. All this requires semantic analysis of regulatory document content.

The technology and know how TokenMill has allows us to build semantic search engines capable of addressing the enterprise search relevance problem. We build semantic networks mapping organization’s language to the one used by the group. Semantic network also overlays organization’s internal processes with available documents so that documents can be connected to company’s activities. This allows us to properly translate user’s query to information retrieval request as well as rank the documents by their correspondence to established processes and protocols the organization is using.

For example the gold mining industry publishes reports on gold extraction details. These include mine layout descriptions, water consumption rates, depth, commodity densities, regulations applicable in the given situation and many other details. All provided in a descriptive text. The facts in those reports and their interrelations impact directly the market's valuation of mining companies. Similarly, medical reports on treatment of various diseases contain descriptions of symptoms, drugs, disease stages. All of this is presented in an unstructured narative form. Yet when it comes to processing vast multitudes of such documents with the aim of extracting common patterns, a manual analysis done by human beings falls short.

Language might appear as too nuanced and too complicated for a machine to understand. At TokenMill we build domain specific text understanding systems capable of extracting meaningful patterns. We use combine tools of artificial intelligence and semantic rules to extract various facts, relationships and patterns from documents in a given domain. Thus the unmanageable flow of textual information is turned into valuable insights.

Different annotation tasks are best done with the help of different tools - annotating each word in a text is very different from labeling the whole article as belonging to a certain class. What constitutes an element to be annotated is another issue. Does an expression “Vilnius - London flight” mention two locations or is it a single entity, that of the route? If one is tasked with annotating opinions then coming up with a definitive judgment might not be so straightforward.

To overcome those difficulties at TokenMill we are employing different annotation tools for different tasks. We use annotation guidelines to help annotators consistently tag elements of the text. As a result we are building quality corpuses of annotated texts which we can then use for machine learning tasks or as a high quality reference point for algorithm evaluation. In cases where no automation is possible, we provide clients with annotated texts as a final result of unstructured data analysis.

Take a simple example of a weather report. At a given date we might have the following weather data points: temperature = +32C, precipitation = 0%, humidity = 30%, wind = 4m/s. This can be taken in as the input for the text generation system and it would spit out something like [ital] “It’s very hot outside, stay in the shade as much as possible. Leave your umbrella at home and pack a bottle of water instead.” Text like that with slight variations could be generated for each different user depending on his or hers preferences and each slight change in weather data points will yield slightly different text. Or take massively multiplayer online games (fantasy sports is another case) with hundreds if not thousands of battles and encounters happening in its vast universe. Battle reports could be generated for fans and spectators of such games. Finance, engineering and other disciplines have cases where a narrative better conveys a message than a chart or a table.