WO2013156828A1 - Method and system for creating and sharing interactive video portraits - Google Patents

Method and system for creating and sharing interactive video portraits Download PDF

Info

Publication number
WO2013156828A1
WO2013156828A1 PCT/IB2012/054590 IB2012054590W WO2013156828A1 WO 2013156828 A1 WO2013156828 A1 WO 2013156828A1 IB 2012054590 W IB2012054590 W IB 2012054590W WO 2013156828 A1 WO2013156828 A1 WO 2013156828A1
Authority
WO
WIPO (PCT)
Prior art keywords
subject
video
videos
interview
user
Prior art date
Application number
PCT/IB2012/054590
Other languages
French (fr)
Inventor
François GIARD
Original Assignee
Talkalter Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Talkalter Inc. filed Critical Talkalter Inc.
Publication of WO2013156828A1 publication Critical patent/WO2013156828A1/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/27Server based end-user applications
    • H04N21/274Storing end-user multimedia data in response to end-user request, e.g. network recorder
    • H04N21/2743Video hosting of uploaded data from client
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • GPHYSICS
    • G11INFORMATION STORAGE
    • G11BINFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
    • G11B27/00Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
    • G11B27/02Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
    • G11B27/031Electronic editing of digitised analogue information signals, e.g. audio or video signals
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47End-user applications
    • H04N21/475End-user interface for inputting end-user data, e.g. personal identification number [PIN], preference data
    • H04N21/4758End-user interface for inputting end-user data, e.g. personal identification number [PIN], preference data for providing answers, e.g. voting
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/85Assembly of content; Generation of multimedia applications
    • H04N21/854Content authoring
    • H04N21/8549Creating video summaries, e.g. movie trailer

Definitions

  • the present invention relates to the field of video portraits, and more particularly to interactive video portraits.
  • Video biographies are usually produced by a team of interviewers with the help of a camera man. Specialized techniques and equipment (lights, lens, sequence editing, storyboard, direction, etc.) are usually required for recording the biography videos.
  • the biography videos are vulnerable due to the support on which they are stored and they need to be manually uploaded to a web server in order to be shared with relatives. Furthermore, the biography videos are expensive to produce since they involve professional workers.
  • the biography videos do not reflect the particular needs of a person since they are produced using a predefined structure for the storyboard.
  • a video interview is linear and the time pace media comes with many constraints. One of the major constraints is the length of the video itself which usually means watching long stretches of non-interactive data. Accessing and identifying relevant information may be difficult.
  • Some software tools allow to keep track of lifetime stories. Lifetime stories are gathered in a random way and are therefore difficult to be retrieved. For example, FacebookTM could be useful to store time-stamped videos about a person but information stored thereon can be trivial and would not be of help to answer questions about the person from future generations.
  • a method for creating an interactive video portrait of a subject comprising: receiving a structured questionnaire for the subject, the structured questionnaire comprising interview questions organized according to a given structure; transmitting the interview questions to the subject; receiving subject videos from the subject, each one to the subject videos comprising an answer to a respective interview question; analyzing and indexing the subject videos to generate the interactive video portrait of the subject; and storing the interactive video portrait.
  • the step of transmitting the interview questions comprises transmitting the interview questions in a video format.
  • the method further comprises a step of creating an interview video for each one of the interview questions.
  • the step of analyzing the subject videos comprises applying a semantic analysis to the subject videos and determining index keywords using the result of the semantic analysis, the subject videos being indexed using the determined index keywords.
  • the method further comprises a step of applying a video analysis to the subject videos and determining additional keywords using a result of the video analysis, the subject videos being indexed using the index keywords and the additional keywords.
  • a system for creating an interactive video portrait of a subject comprising a processing unit coupled to a storing unit, the processing unit being configured to execute the steps of the above described method.
  • a method for providing an answer to a question of a user about a subject via an interactive video portrait of the subject comprising: providing the user with an access to the interactive video portrait of the subject, the interactive video portrait comprising a plurality of videos answers of the subject being organized according to a given structure and each having index keywords associated thereto; receiving the question from the user; retrieving at least one of the plurality of videos that corresponds to an answer to the question of the user; and sending the retrieved video to the user.
  • the step of receiving the question comprises receiving a user video containing the question.
  • the retrieving step comprises applying a semantic analysis to the question and determining search keywords using a result of the semantic analysis.
  • the retrieving step further comprises identifying at least one of the video answers having at least one keyword associated thereto that relates to at least one of the determined search keywords.
  • a system for creating an interactive video portrait of a subject comprising: a questionnaire unit adapted to receive a structured questionnaire for the subject, the structured questionnaire comprising interview questions organized according to a givens structure; a subject communication unit adapted to transmit the interview questions to the subject and receive subject videos from the subject, each one to the subject videos comprising an answer to a respective one of the interview questions; and an indexing unit adapted to analyze and index the subject videos to generate the interactive video portrait of the subject, and store the interactive video portrait.
  • the subject communication unit is adapted to transmit the interview questions in a video format.
  • the system further comprises an interview unit adapted to generate an interview video for each one of the interview questions;
  • the indexing unit is adapted to apply a semantic analysis to the subject videos and determine index keywords using the result of the semantic analysis, the subject videos being indexed using the determined index keywords.
  • the indexing unit is further adapted to apply a video analysis to the subject videos and determine additional keywords using a result of the video analysis, the subject videos being indexed using the index keywords and the additional keywords.
  • a system for providing an answer to a question of a user about a subject via an interactive video portrait of the subject comprising: a providing unit for providing the user with an access to the interactive video portrait of the subject, the interactive video portrait comprising a plurality of videos of the user being organized according to a given structure and each having index keywords associated thereto; a retrieving unit for receiving the question from the user and retrieving at least one of the plurality of videos that corresponds or relates to an answer to the question of the user; and a transmission unit for sending the retrieved video to the user.
  • the retrieving unit is adapted to receive a user video containing the question.
  • the retrieving unit is adapted to apply a semantic analysis to the question and determine search keywords using a result of the semantic analysis.
  • the retrieving unit is further adapted to identify at least one of the video answers having at least one keyword associated thereto that relates to at least one of the determined search keywords.
  • FIG. 1 is a flow chart of a method for creating an interactive video portrait, in accordance with an embodiment
  • FIG. 2 is a flow chart of a method for retrieving videos contained in the interactive video portrait of FIG. 1 ;
  • FIG. 3 is a block diagram of a system for creating an interactive video portrait and providing users with access to the interactive video portrait, in accordance with an embodiment
  • FIG. 4 is a block diagram illustrating an exemplary architecture for the system of FIG. 3.
  • FIG. 5 illustrates an exemplary interface provided to a subject for recording subject videos.
  • the interactive video portrait comprises a set of videos of the subject in which he tells stories and/or answers questions on various topics of interest.
  • the interactive video portrait is stored on a web platform and users such as relatives or members of a group of interest may access the interactive video portrait and optionally ask further questions.
  • the users may query on various topics of interest via the interactive video portrait interface.
  • the present method and system automates the process of collecting, preserving and sharing the interactive video portraits of subjects.
  • a personalized structured questionnaire is created for each subject and the structure questionnaire is presented to the subject through interview videos.
  • the present method and system automates the video interview using speech recognition and advanced video tracking and detection tools.
  • FIG. 1 illustrates one embodiment of a method 100 for creating an interactive video portrait for a subject.
  • basic information about the subject is collected.
  • the basic information collected at step 102 comprises information necessary for creating a subject profile. Questions such as binary questions are presented to the subject who provides an answer for each question. Examples of questions comprise questions about the subject's age, the subject's marital status, whether the subject has children, the subject's areas of interest, the subject's profession, etc.
  • the subject is guided in the process of creating a genealogy at step 102.
  • the subject may be presented with an empty genealogy tree and may complete the empty genealogy tree by providing information about his parents, sisters, brothers, children, uncles, cousins, and/or the like.
  • the information may comprise the first name, the last name, contact information such as an email address for example, and/or the like for each contact person included in the genealogy tree.
  • a subject profile is generated for the subject at step 102.
  • the subject profile may further comprise subject authentication information, contact information, pictures of the subject, and/or the like.
  • the information collected at step 102 may be stored in the subject profile.
  • the subject profile may further comprise access information that identifies users who may have access to the subject interactive video portrait, and the parts of the interactive video portrait that an authorized user may access.
  • access information may comprise an identification for members of a group of interest, users, etc.
  • members of a group or users may be associated with a time delay so that permissions can be granted at a specific date.
  • a structured questionnaire is created for the subject based on the personal information collected at step 102.
  • the structured questionnaire is a sequence of interview questions designed to obtain statistically useful information about given topics for the subject.
  • the content and sequence order of the interview questions are determined using the user profile and preference in order to collect the desired information about the subject. For example, if method 100 is used for collecting information about the life of the subject, then the structured questionnaire may take on the style of a typical biographer questionnaire and may use a structure that systematically covers each area of the subject's biography.
  • the information collected at step 102 may be used for determining a priority in the flow of interview questions to be presented to the subject.
  • the structured questionnaire is designed for collecting information about the life of a subject.
  • the structured questionnaire includes interview questions that may be organized in themes such as family, activities, vacations, sports, etc.
  • a respective structured questionnaire is created for each subject so that the structured questionnaire may vary from one subject to another.
  • each structured questionnaire is specific to a given subject and the subject- specific structured questionnaire is created using the subject information collected at step 102.
  • a single structured questionnaire is created once at step 104 and the same structured questionnaire applies for all of the subjects.
  • the subjects are then presented with the same structured questionnaire and only respond to questions that are relevant to them. For example, if the subject does not have children, he may skip the interview questions about children.
  • a single generic structured questionnaire is created once at step 104 and the method 100 comprises a further step of adapting or personalizing the generic structured questionnaire for each subject.
  • the generic structured questionnaire is personalized for each subject using the subject personal information collected at step 102.
  • the personalization of the generic structured questionnaire may be performed by generating a subject-specific path connecting at least some of the questions included in the generic structured questionnaire. For each subject, the path identifies the interview questions included in the generic structured questionnaire to be asked to the subject and/or defines the order in which the identified questions should be presented to the subject. For example, if the subject has specified that he is childless and never got married, then the subject-specific path does not connect interview questions related to spouses and children so that these interview questions are disabled and will not be presented to the subject.
  • An expert may create the interview questions on a topic of interest and create interrelation paths between the interview questions so as to structure the interview questions.
  • the paths between the interview questions may regroup the interview questions per topics or themes, indicate the sequence that should be followed when asking the interview questions, indicate a temporal order for the interview questions, and/or the like.
  • the structured questionnaire may not include interview questions about children.
  • the structured questionnaire may comprise interview questions about children that would be different from the interview questions that would be asked to a subject who has children. For example, a childless subject may be asked to give the reasons why that is so while a subject having children may be asked to discuss about the preferred activities of his children.
  • the structured questionnaire may comprise different versions of a same question and the version of the question that is suitable for the subject is selected using the information collected at step 102.
  • the suitable version of a given question may be selected as a function of the age of the subject.
  • the expert may assign at least one index keyword to each interview question contained in the structured questionnaire.
  • the index keywords allow to retrieve the answers according to the questions asked as opposed to retrieving those indexed by semantic analysis.
  • the structured questionnaire is organized as a tree where the branches each represent themes such as family, friends, work, etc., and the leaves each represent interview questions.
  • the structured questionnaire is provided to the subject for validation.
  • the subject may then deactivate or skip interview questions included in the structured questionnaire.
  • the subject may also assign access information to given interview answers included in his interactive video portrait in order to prevent some users from accessing the subject videos corresponding to the given interview questions.
  • the expert may further create multiple structured questionnaire that may be used by corporations, marketing agencies, focus group researchers, interest group managers, clinician report personnel, etc.
  • an interview video is created at step 106 for each interview question included in the structured questionnaire that has been created at step 104.
  • a person such as an actor for example, records an interview video in which he asks the corresponding interview question.
  • the interview videos are automatically generated by appropriate software/hardware.
  • the interview videos each contain a computer- generated avatar who asks a respective responding interview questions.
  • the subject may be presented with a choice of actors/avatars and may select a desired actor/avatar for the creation of the interview videos.
  • interview videos are linked together according to the same structure as that of the structured questionnaire.
  • the interview videos are presented to the subject who records videos of himself, for example subject videos, while answering the interview questions.
  • a subject video is created for each interview question at step 110. It should be understood that the subject may decide not to answer all of the questions contained in the structured questionnaire or may record more than one subject video for a same interview question.
  • the interview videos are automatically played back according to the structure of the structured questionnaire.
  • a second interview question video is automatically played back.
  • a third interview video is automatically played back, etc.
  • the structured questionnaire is provided to the subject who may modify the order in which he would like to answer the interview questions. As described above, the subject may also skip some questions and/or add additional questions. In the latter case, an interview video is created for the additional interview questions.
  • the subject may assign at least one keyword for at least some of the subject videos.
  • the subject may also assign localization information such as an address or a location name to a subject video in order to permit subsequent identification on a map.
  • the subject may also assign contact persons for at least some of the subject videos in order to permit or prevent identified contact persons from accessing the given videos. The contact persons that have been authorized to have access to the given subject videos may be subsequently notified that the given videos are available.
  • the subject may further attach media content to the subject videos.
  • media content may help illustrate the answer to an interview question given in a subject video.
  • a plurality of interview videos are displayed to the subject for a same interview question. For example, a first interview video in which an actor or an avatar asks a question is first presented to the subject, and a second interview video is presented to the subject while he is answering the question asked in the first interview video.
  • the second interview video may present the actor or avatar in an idle state in which he listens to the answer given by the subject. In this case, the second interview video is referred to as an idle video.
  • the subject video is streamed in substantially real-time to the server while being recorded, and automatically analyzed to determine which idle video is to be presented to the subject. For example, audio and video analysis may be applied to the subject video to determine the emotional state of the subject while answering the interview question. For example, if smiles or laughs are detected in the subject video, then an idle video in which the actor/avatar smiles may be presented to the subject. In another example, if it is determined that the subject is sad while answering the interview question, then an idle video in which the actor/avatar appears to be compassionately listening is presented. In a further example, if hesitations, surprises, grimaces, etc.
  • an idle video in which the actor/avatar has a neutral attitude may be presented to the subject.
  • the server may have stored thereon idle videos each presenting the actor/avatar in a different idle state, and the appropriate idle video to be presented to the subject is determined via video and/or audio analysis of the subject video.
  • video analysis may be used to determine an answer to an interview question.
  • an interview video in which an actor/avatar asks whether the subject would like to answer questions about politics may be presented to the subject.
  • the subject may answer the question by shaking his head.
  • video analysis is performed to extract the movement of the subject's head. If it is determined that the subject shakes his head from left to right, then it is determined that the subject does not want to discuss about politics and the subject is presented with interview videos related to other topics. If it is determined that the subject shakes his head from top to bottom, then it is determined that the subject accepts to discuss about politics and the subject is presented with interview videos related to politics.
  • non-verbal communication may be affected by the culture and the environment in which the subject was raised and/or lived for a substantial amount of time.
  • a subject of Indian descent may use a different head shake, such as a head bobble, to indicate an acceptance of the question or an encouragement for the actor/avatar to continue with the process.
  • the video analysis may use this data. If data is unavailable and/or video analysis does not allow a definitive decision on the meaning of the non-verbal gesture, a video may be presented to the subject in which the actor/avatar requests a specific verbal answer.
  • video analysis of the streamed subject video may be performed for different purposes. For example, it may be determined that the subject is no longer in the field of view of the camera, the person in front of the camera is not the subject (using face recognition methods), the subject is not moving, the eyes of the subject are closed (and the subject may be sleeping), etc.
  • the length of the subject video is determined and compared to an expected video length, such as an average video length, or a range of video lengths. The subject may be informed about whether the subject video has a "normal" length, for example a length that substantially corresponds to the expected length or is comprised within the range of video lengths.
  • the subject may also be informed whether the subject video is shorter or longer than the expected video length.
  • the subject video in which the subject video is streamed in substantially real-time to the server, the subject may be informed in substantially real-time when the length of the subject video he is recording exceeds the "normal" length or the range of video lengths.
  • the interview questions are presented to the subject via interview videos, it should be understood that the interview questions may be presented in a written form.
  • step 106 is omitted and step 108 comprises a step of presenting to the subject the interview questions in a written form, which are displayed on the subject display unit.
  • the interview questions may be presented in an audio-only media.
  • step 106 includes the steps of recording oral interview questions and storing the audio files containing the oral interview questions
  • step 108 includes the steps of transmitting the audio files to the subject and playing back the audio files on the subject machine to present the oral interview questions to the subject.
  • interview questions may be presented to the subject in a written form while other interview questions may be presented via interview videos.
  • the subject videos are first analyzed and then indexed and organized according to index keywords.
  • Natural speech recognition is applied to the audio track of each subject video in order to translate the spoken words contained in the audio track into text. It should be understood that any adequate method of natural speech recognition may be used. For example, Dragon NaturallySpeakingTM software may be used.
  • semantic analysis is run over the text corresponding to the audio track of the subject video to generate a list of index keywords.
  • the list of index keywords comprises at least one theme/topic that corresponds to the subject video.
  • video analysis may also be run over the video tracks to generate a list of facial feature keywords. For example, 30 to 60 feature facial points may be tracked to extract statistics information and interpret facial expressions.
  • Software such as FaceAPITM or FaceReaderTM may be used for the video analysis.
  • a list of corresponding keywords is generated using the keywords determined by the semantic analysis, the keywords assigned to the corresponding interview questions, and, if any, the keywords assigned to the interview video by the subject and the keywords determined from the video analysis. The determined keywords are assigned to the respective subject video.
  • the semantic analysis is performed by a recognition engine based on finite-state transducers.
  • the speech recognition engine applies a Large Vocabulary Continuous Speech recognition model that receives the audio track as input and outputs a lattice of assumptions, a language model for Large Vocabulary Continuous Speech having a conversational style, a pronunciation model, an acoustic model such as a Hidden Markov model (HMM) or Linear Discriminant Analysis (LDA)/ Likehood Linear Transform (LLT)/ Feature Maximum Likehood Linear Regression (fMLLR) transform matrices, etc.
  • HMM Hidden Markov model
  • LDA Linear Discriminant Analysis
  • LLT Likehood Linear Transform
  • fMLLR Feature Maximum Likehood Linear Regression
  • the whole written text corresponding to the audio track of the subject video is also stored and assigned to the subject video.
  • the natural speech recognition may not be able to recognize and convert into text all of the words or expressions contained in the audio track of the subject video.
  • the audio wave for example the phonetic signature
  • the phonetic signature may be extracted using the KaldiTM speech recognition toolkit for example.
  • the subject may interact with the system, e.g. he may answers to questions saying "yes" or "no".
  • the speech recognition engine may be further adapted to detect vocal activity from the subject, reduce noise and echo, and use specific speech recognition grammar that may include rejection mechanisms.
  • the keywords assigned to a subject interview are categorized.
  • the keywords assigned to interview questions and the keywords assigned by the subject form a list of first-level priority keywords while the keywords generated by the semantic analysis are included in a second-level priority list. This allows users to find answers about specific words/topic in non-expected videos. For example, if a user queries the word "fish", the system will prioritize the detected "fish” words and related semantic themes such as "sports", "trout”, etc.
  • the list of determined index keywords is provided to the subject who may add additional keywords.
  • the subject may be provided with a list of selectable keywords generated by semantic analysis based on the detected keywords from the text of the subject video, and the subject may manually deactivate unsuitable keywords which otherwise would be assigned to the respective subject video along with the automatically generated index keywords previously assigned to the subject video.
  • index keywords may also be assigned to the content media.
  • a picture assigned to a given subject video by the subject may have assigned thereto the same index keywords as those that have been generated for the corresponding subject video.
  • names of contacts cited by the subject in a given subject video are detected.
  • au automatic notification may be sent to the identified contact for informing the contact that the given subject video is available for play back.
  • a confirmation request may be provided to the subject for permitting the cited contact to have access to the given subject video. The subject may confirm or not the permission. If permitted by the subject, a notification may also be sent to the contact.
  • the subject receives a notification to enter contact information such as an email address in order to send an invitation to the detected contact.
  • the emotional state of the subject is also determined for each subject video.
  • An expression recognition analysis is performed on the video and/or audio tracks of the subject video in order to determine the emotional state of the subject while recording the subject video. For example, laughs may be identified within the audio tracks, smiles may be detected from the video tracks in order to determine whether the subject is happy or sad.
  • Basic statistics such as agitation, head position, mouth and eye status, etc. can also be retrieved from the video analysis.
  • the determined emotional state may then be tagged to the corresponding subject video.
  • the emotional state may correspond to an index keyword assigned to the subject video.
  • the system can be used to detect patterns in emotional state.
  • the subject video may be analyzed to perform a non-verbal evaluation.
  • Elements such as head agitation frequency, body agitation frequency, eye movement frequency, eye direction statistics, voice pitch variation, speech speed variation, skin color variation, emotion recognition, hesitation statistics, and/or the like may be determined from the video and tagged to the subject video.
  • the determined non-verbal evaluation elements may be stored in metadata assigned to the subject video.
  • the semantic analysis performed on the subject videos is added to the subject profile.
  • the structured questionnaire is updated using the results of the semantic analysis. For example, new areas of interest for the subject may be detected using the results of the semantic analysis and questions related to the new areas of interest may be inserted in the structured questionnaire of the subject.
  • interview questions that were previously disabled may be included in the subject-specific path.
  • interview videos may be generated for the newly inserted interview questions and subsequently presented to the subject.
  • the structure of the structured questionnaire may also be modified according to the results of the semantic analysis. For example, if specific interests in sport are detected in the semantic analysis of previous answers, then the Sports topic section of the questionnaire will be raised in the priority list of the questions presented to the subject.
  • the indexed subject videos forming an interactive video portrait for the subject are stored in memory at step 114, for example each subject video and its assigned index keywords are stored in the memory.
  • the interactive video portrait of a given subject may then be accessed by users who may play back the subject videos corresponding to the given subject.
  • the structured questionnaire of the given subject may be provided to the users, for example the list of the interview questions contained in the structured questionnaire may be displayed to the users who may select a given interview question and then access the subject video corresponding to the given interview question.
  • the interactive video portrait may also allow a virtual live conversation between a given subject and a user.
  • the user connects to a system on which the interactive video portrait of the given subject is stored and asks a specific question.
  • the system retrieves at least one subject video that corresponds to an answer to the specific question asked by the user, and the retrieved subject video is played back to the user.
  • FIG. 2 illustrates one embodiment of a method 120 that allows such a virtual live conversation between a user and a given subject.
  • a specific user question about a given subject is received from a user. If there is no access restriction associated with the interactive video portrait of the given subject, then any user may access the interactive video portrait. Alternatively, only the users that have been authorized by the given subject may access the interactive video portrait.
  • the method comprises a step of providing the user with a list of questions that may be virtually asked to the given subject.
  • the user selects a desired question from the list.
  • the questions included in the list may correspond to interview questions that have been asked to the given subject while creating his interactive video portrait. It should be understood that the list of selectable questions provided to the user may comprise only the interview questions that were answered in the structured questionnaire by the given subject.
  • the user may ask his own question.
  • the user question may be a written question entered by the user via an input device such as a keyboard.
  • the user question may be asked orally via a microphone.
  • the user may create a video containing a video and an audio tracks, in which he asks a specific question to the given subject.
  • natural speech recognition is used for converting the audio track of the user video into text.
  • the user may enter a text query to ask a specific question to the given subject.
  • the user may use a combination of text, audio and video segments to formulate his question to the given subject.
  • At least one subject video is identified from the interactive video portrait of the given subject at step 124.
  • the identified subject videos correspond to potential answers from the given subject to the user question received from the user.
  • search keywords are first determined from the received question using semantic analysis, as described above with respect to subject videos.
  • Search keywords such as search semantic themes/topics are extracted from the semantic analysis.
  • the semantic themes and the detected search keywords are then extrapolated with synonym groups.
  • the indexation gathered from the questions and the analyzed answers is used to find related themes derived from synonyms keywords and the determined related themes are ranked by relation frequency. For example, a user may ask a question about sailing. In this case, keywords related to "sailing" such as sail, regatta, water sport, wind, boat, and the like, are first determined.
  • the system will first retrieve and play subject videos that are related to the sport section of the structured questionnaire, and then retrieve all subject videos having keywords corresponding to the determined related keywords whether these subject videos are contained in the sport section of the structured questionnaire or not.
  • the user question is an oral question contained in an audio file
  • a step of speech recognition is first performed in order to convert the oral question into text from which search keywords can be determined.
  • the received user question is an oral question contained in an audio track of a video
  • a step of speech recognition is first performed in order to convert the oral question into text from which search keywords can be determined.
  • the search keywords relevant to the index keywords and themes assigned to the subject videos, and the subject videos having at least one index keyword that corresponds to at least one search keyword may be identified as a potential answer to the received user question.
  • the identified subject videos are then provided to the user at step 124.
  • the natural speech recognition may not be able to recognize and convert into text all of the words or expressions contained in the audio track of the user question video.
  • the audio wave for example the phonetic signature
  • the phonetic signatures assigned to the subject videos are compared to the phonetic signatures assigned to the subject videos in order to find potential matches.
  • the subject videos provided to the user are automatically played back on the subject machine.
  • the list of retrieved subject videos is provided to the user and an indication of their relevance may also be provided.
  • only the most relevant subject video is automatically played back on the user machine and the other retrieved subject videos may be played back upon selection by the user. In this case, the user experiences a substantially live virtual conversation with the subject. Each time the user asks a question, a subject video in which the subject provided an answer to the question is automatically played back on the user machine.
  • all of the subject videos having at least one index keyword that matches/relates to one of the search keywords are considered as potential answers and are provided to the user.
  • only the subject videos having at least two keywords that match search keywords are considered as potential answers and are provided to the user.
  • the subject videos that are provided to the user are ranked as a function of their relevance relative to the user question. The ranking may be calculated based on the number of matches between search keywords index keywords and their relative theme group, the occurrence frequency of the search keywords, and/or the like.
  • the determined search keywords are provided to the user before the determination of the subject videos that correspond to answers to the user question.
  • the user may update the list of search keywords by adding/removing search keywords and the identification of the suitable subject videos is then performed based on the updated search keywords.
  • an emotional state indication is assigned to the subject videos
  • the emotional state indication may also be provided to the user at step 124 along with the determined subject videos.
  • media content is also provided to the user at step 124 along with the determined subject videos.
  • users may assign an appreciation indication to subject videos. In this case, the appreciation indication may be provided to the user along with the respective subject video.
  • users can mark a given subject video as inappropriate if content found therein is offending.
  • the given subject video may be put in quarantine until a decision from an administrator is made.
  • users may also send comments and/or a testimony after play back of a given subject video, and a notification may be sent to the subject.
  • the subject videos are organized according to the same structure as that of the structured questionnaire. If the subject videos are organized as a tree, a user may select a specific branch of the subject video tree representing a specific topic or theme in order to specify a search criteria. In this case, the user is provided with the subject videos related to the selected branch. [00102] In one embodiment, the user may found answers of a given subject by directly browsing the structured subject videos for the given subject.
  • a user may ask an additional interview question to the subject.
  • the user creates an interview video in which he asks the additional interview question.
  • the subject is then notified that an additional interview video has been posted by the user.
  • FIG. 3 illustrates one embodiment of a system 130 adapted to execute the methods 110 and 120.
  • the system comprises a server 132, a subject machine 134, an expert machine 136, and a user machine 138, which are each provided with at least a processing unit coupled to a storing unit for storing data thereon and a communication unit.
  • the subject machine 134, the expert machine 136, and the user machine 138 are connected to the server 132 via a communication network 140.
  • the server 132 is adapted to receive subject information from the subject machine 134 that is further provided with an input device such as a keyboard, a sound speaker, a microphone and a video camera.
  • the server 132 is adapted to transmit the structured questionnaire from the expert machine 136 which is provided with a display and an input device such as a keyboard.
  • An expert can generate a structured questionnaire containing interview questions as described above using the expert machine 136.
  • the expert machine 136 transmits the structured questionnaire to the server 132.
  • the server 132 is adapted to transmit the interview questions in a written form to the subject machine 134. The interview questions are then displayed on the display unit of the subject machine 134.
  • the server 132 is configured for generating interview videos based on the received structured questionnaire.
  • Each generated interview video comprises an avatar asking a corresponding interview question.
  • the system 130 further comprises an actor machine 142 provided with at least a processing unit coupled to a storing unit, a communication unit, a display unit, a video camera such as a webcam for example, and a microphone.
  • the structured questionnaire is sent to the actor machine from the server 132 and an actor records an interview video for each one of the interview questions contained in the structured questionnaire.
  • the interview videos are then sent to the server 132.
  • the server 132 is further configured for indexing the received interview videos and transmit the indexed interview videos to the subject machine 134, as described above.
  • the server 132 is also configured for receiving a user question from the user machine 138 that is provided with an input device, a sound speaker, a display unit, and optionally a microphone and/or a video camera.
  • the server 132 is adapted to retrieve subject videos corresponding to the user questions, as described above. The server 132 then sends the retrieved subject videos to the user machine 138.
  • the interview questions may also be presented to the subject in an oral-only form.
  • the server 132 may be adapted to automatically generate oral questions that are stored in audio files.
  • the oral questions may be recorded by an actor via an actor machine 142 and then uploaded to the server 132.
  • the server 132 then transmits the audio files to the subject machine 134.
  • the user machine 138 is configured for automatically play back at least one of the received subject videos, as described above.
  • FIG. 4 illustrates one exemplary architecture for the server 132.
  • the server 132 comprises a subject profile unit 150, a questionnaire unit 152, an interview video unit 154, a video providing unit 156, a video indexing unit 158, a video retrieving unit 160, a notification unit 162, and a database 164.
  • the subject profile unit 150 is adapted to retrieve questions from the database 164, transmit the questions to the subject machine 134, and receive answers to the questions from the subject machine 134. Questions from the structured questionnaire may be sent to the subject who provides an answer for each question. Examples of questions comprise questions about his age, his genealogy, his friends, etc. [00114] In one embodiment, the subject profile unit 150 is adapted to guide the subject in the process of creating a genealogy tree. For example, the subject profile unit 150 may be adapted to transmit an empty genealogy tree and the subject may complete the empty genealogy tree by providing information about his parents, sisters, brothers, children, uncles, cousins, and/or the like. The information may comprise the first name, the last name, contact information such as email address for example, and/or the like for each person included in the genealogy tree. The completed genealogy tree is then sent to the subject profile unit 150 and stored in the database.
  • the subject profile unit 150 is further adapted to generate a subject profile for each subject.
  • the user profile may comprise subject identification information, contact information, pictures of the subject, and/or the like.
  • the subject profile may further comprise access information.
  • the subject profile unit 150 is adapted to recognize human faces in order to identify the subject using a picture or video received from the subject machine 134. Face recognition or voice recognition may also identify users permissions over specific subject videos.
  • the questionnaire unit 152 is adapted to retrieve the subject information from the database 164 and transmit the subject information to the expert machine 136.
  • An expert creates a structured questionnaire which is transmitted to the questionnaire unit 152 from the expert machine 136 and stored in the database 164.
  • the structured questionnaire contains interview questions interrelated by paths to form a structure, and index keywords for each interview question.
  • the structured questionnaire is organized as a tree where the branches each represent topics such as family, friends, work, etc., and the leaves each represent interview questions.
  • the questionnaire unit may be adapted to provide a tree structure to the expert machine that is used for assigning interview questions to the leaves of the tree, assigning topics to the branches of the tree, adding/removing branches and leaves, etc.
  • the questionnaire unit 152 is adapted to transmit the structured questionnaire to the subject machine 134 for validation by the subject. The subject may then delete interview questions included in the structured questionnaire and/or add additional interview questions in order to further personalize the structure questionnaire. The subject may also assign access information to given interview questions included in the structure questionnaire in order to prevent some users from accessing the subject videos corresponding to the given interview questions.
  • the interview video unit is adapted to retrieve the structured questionnaire from the database and transmit the structured questionnaire to the actor machine 142. For each interview question included in the structured questionnaire, a person, such as an actor for example, records an interview video in which he asks the corresponding interview question. The interview videos are then sent to the interview video unit 154 and stored in the database 164. It should be understood that the interview videos are organized according to the structure of the structured questionnaire.
  • the interview video unit 154 is adapted to generate interview videos in which an avatar asks a respective interview question. The generated interview videos are stored in the database 164. In this case, the interview unit 154 may be adapted to send a list of avatar to the subject machine 134 and receive an indication of a desired avatar from the subject machine 134. Then, the interview videos are generated using the selected avatar.
  • the video providing unit 156 is adapted to retrieve the interview videos from the database 164 and transmit the interview videos to the subject machine 134.
  • the subject machine is adapted to play back the interview videos and record a subject video in which the subject answers the question asked by the actor or avatar in the interview video.
  • the subject videos are sent from the subject machine 134 to the video indexing unit 158.
  • the video providing unit 156 is adapted to sequentially transmit the interview videos to the subject machine 134 and automatically play back the interview videos on the subject machine 134.
  • the video providing unit 156 is adapted to transmit the structured questionnaire to the subject machine 134. The subject may then select a given interview question and an indication of the selected interview question is transmitted to the video providing unit 156 which retrieves the corresponding interview video and transmits it to the subject machine 134.
  • the subject may create an additional interview question which is sent to the interview video unit 154 so that a corresponding interview video be created.
  • the video indexing unit 156 is adapted to receive the subject videos from the subject machine 134 and organize and index the subject videos based on index keywords.
  • the video indexing unit is adapted to perform natural speech recognition on the audio track of each subject video in order to translate the spoken words contained in the audio track into text, as described above.
  • the video indexing unit is further adapted to perform semantic analysis in order to determine index keywords and themes from the text of the subject videos, the structure of the structured questionnaire, the keywords assigned to the corresponding interview questions, and, if any, the keywords assigned to the interview video by the subject, as described above.
  • the determined keywords are assigned to the respective subject video and stored in the database 164.
  • the video indexing unit 156 may further be adapted to determine phonetic signatures and assigned the determined phonetic signatures to their respective subject videos, as described above.
  • the subject may assign at least one keyword to an interview video and the assigned keyword is transmitted to the video indexing unit 158.
  • the subject may also assign a contact person to a subject video in order to permit or prevent the identified contact persons from accessing the given videos.
  • the subject may also assign location information to a subject video in order to access the given videos using geolocalisation.
  • An indication of the contact person is then transmitted to the notification unit 162 that retrieves the contact information such as the email address of the identified contact person and transmits a notification that a subject video of the subject is available.
  • the video indexing unit 156 is adapted to transmit the list of determined index keywords to the subject machine 134 so that the subject who may add/remove additional keyword.
  • the modification to the list of index keywords are transmitted to the video indexing unit 156 that updates the list of keywords accordingly.
  • the subject may further attach media content to a subject video. For example, pictures, text, additional videos, and/or the like may be attached to a given subject video. Media content may help illustrate the answer to an interview question given in a subject video.
  • the media content is then transmitted to the video indexing unit 158 and stored in the database 164.
  • index keywords may also be assigned to the content media. In this case, the index keywords are transmitted to the video indexing unit 158 along with the media content.
  • the video indexing unit 158 is adapted to recognize names of contact persons contained in the subject videos using the natural speech recognition. The names of the contact persons is then transmitted to the notification unit which is adapted to retrieve the contact information of the identified contact persons and send them a notification that a subject interview is available. Before sending the notification, the notification unit 162 may request a confirmation from the subject.
  • the video indexing unit 158 is adapted to recognize names of places like city names for example contained in the subject videos using the natural speech recognition. The names of the places are then attached to the corresponding subject videos.
  • the video indexing unit 158 is adapted to determine the emotional state of the subject for each subject video and the determined emotional state is assigned to the corresponding subject video.
  • the video indexing unit 158 is adapted to perform a non-verbal evaluation of each subject video.
  • the video indexing unit 158 is then adapted to determine elements such as head agitation frequency, body agitation frequency, eye movement frequency, eye direction statistics, voice pitch variation, speech speed variation, skin color variation, emotion recognition, and/or the like may be determined from the video and tagged to the subject video. These elements are then assigned to their respective subject video and stored in the database 164.
  • the video retrieving unit 160 is adapted to provide a user access to the interactive video portrait of a subject. In one embodiment, the video retrieving unit 160 is adapted to verify user identification information received from a user machine 138 before allowing the user to access the interactive video portrait.
  • the video retrieving unit 160 is adapted to receive a user question about a given subject from the user machine 138 and determine the subject videos that correspond to an answer to the received user question.
  • the video retrieving unit 160 is adapted to generate a list of questions that may be virtually asked to the given subject and transmit the generated list to the user machine 138.
  • the user selects a desired question from the list and an indication of the selected question is sent to the video retrieving unit 160 which retrieves the subject video that contains an answer to the selected question.
  • the video retrieving unit 160 is adapted to receive a question created by the user.
  • the received user question may be a written question entered by the user via an input device such as a keyboard.
  • the received user question may be asked orally via a microphone.
  • the received question may be contained in the audio track of a received video.
  • the video retrieving unit 160 is adapted to first determine search keywords from the received user question using semantic analysis.
  • the video retrieving unit 160 is further adapted to convert the oral question into text from which search keywords and themes can be determined.
  • the video retrieving unit 160 is further adapted to compare the search keywords to the index keywords assigned to the subject videos.
  • the subject videos having at least one index keyword that corresponds to at least one search keyword or a theme may be identified as potential answers to the received user question.
  • the video retrieving unit 160 then retrieves the identified subject videos and transmit them to the user machine 138.
  • the user machine 138 is adapted to automatically play back the subject videos received from the video retrieving unit 160.
  • the video retrieving unit 160 transmits the list of he retrieved subject videos.
  • the video retrieving unit 160 may also be adapted to determine the relevance of each subject video contained in the list and transmit the determined relevance along with the list.
  • the user machine 138 is adapted to automatically play back only the most relevant subject video while the other retrieved subject videos may be played back upon selection by the user.
  • the user experiences a substantially live virtual conversation with the subject.
  • a subject video in which the subject provided an answer to the question is automatically played back on the user machine.
  • the video retrieving unit 160 is adapted to transmit to the user machine 138 all of the subject videos having at least one index keyword that matches one of the search keywords or a theme group are considered as potential answers and are provided to the user. In another embodiment, only the subject videos having at least two keywords that match search keywords or theme groups are considered as potential answers and are provided to the user, etc.
  • the video retrieving unit 160 is adapted to generate a ranking of the subject videos as a function of their relevance relative to the user question. The ranking may be performed based on the number of match between search keywords and index keywords, the occurrence frequency of the search keywords, and/or the like.
  • the video retrieving unit 160 is adapted to retrieve the emotional state assigned to the subject videos from the database and transmit to the user machine 138 an indication of the detected emotional state along with the determined subject videos.
  • the video retrieving unit 160 is adapted to retrieve the media content assigned to the determined subject videos and transmit it along with the determined subject videos.
  • a user may ask an additional interview question to the subject.
  • the user creates an interview video in which he asks the additional interview question, and the user interview video is sent to the interview video unit 154 and stored in the database 164. The subject is then notified that an additional interview video has been posted by the user.
  • a user may add an additional testimony to the subject video.
  • the user records a testimony video in which he tells the additional interview information, and the user testimony video is uploaded to the interview video unit 154 and stored in the database 164. The subject is then notified that an additional testimony video has been posted by a user.
  • FIG. 5 illustrates an exemplary interface 170 that may be provided to the subject by the server 132, for example, for recording the subject videos.
  • the interface 170 includes a first region or area 172 for displaying the interview questions of the structured questionnaire, a second region 174 for displaying the interview video, a third region 176 for displaying the subject video while it is being recorded, a play back toolbar 178, and a recording toolbar 180.
  • the subject may select a given interview question from the list of questions displayed in the first region 172 and the corresponding interview video is played back in the second region.
  • the play back toolbar 178 is used by the subject to control the play back of the interview video while the recording toolbar 180 is used by the subject for controlling the recording of the subject video.
  • the above-described method and system may be used in a family context to collect, preserve and share intergenerational family memory. It can be used by distant relatives and friends using a rights management system preset. Intergenerational family memory can be collected, preserved and shared across multiple generations.
  • the above-described method and system may be used to collect, preserve and share oral tradition and collective memory. Natives and others can collect, preserve and share oral cultural tradition. Oral societies will be able to keep and preserve their cultural heritage using their oral communication mode.
  • the above-described method and system may be used in the medical field, for health problem detection, for patient pre-diagnosis, for pharmacological tracking, for therapeutic tracking, distance video diagnosis, therapy outcome measures, etc.
  • the above-described system may be adapted to generate and display graphics and charts from video detection tools vectors (emotion state, agitation level, eye move stat, voice pitch, skin color evolution, etc.) so that a clinician can get a quick state overview on a specific question or on the overall interview.
  • the above-described system may generate and display forms and quiz aggregation in the form of charts for example.
  • the above-described method and system may be used in managing focus groups to gather large quantities of answers, for market analysis, for marketing agencies or to collect and manage large social studies databases.
  • focus groups focus groups
  • surveys consumers' opinions can be collected, sorted and analyzed quickly.
  • Questionnaires can be transmitted to large numbers of people over the internet or directly in retail stores.
  • marketing agencies may collect consumers' opinions.
  • the above-described method and system may be used to generate data mining request for market analysis.
  • the above-described method and system may be used in the eLearning industry for knowledge transfer, education and fundamental research.
  • the above-described method and system may be used in the tourism industry by collecting and sharing user experiences with other users in oral mode.
  • the above-described method and system may be used to share local information using geological digital maps found on the Internet.
  • the above-described method and system may be used to share local information using Ground Positioning System (GPS) based tools such as mobile phones.
  • GPS Ground Positioning System
  • the above-described method and system may be used for preparation and screening of candidates in the context of job interviews.
  • the above-described method and system may be used in human resources, for rapid candidate screening, employee tracking and performance review.
  • the system can be used in human resources to facilitate the evaluation of a large number of candidates.
  • the above-described method and system may be used to transmit specialized knowledge of a worker to his replacement.
  • the above-described method and system may be used for example in psychiatry or grading.
  • psychiatry an individual can respond much more easily, and at his own pace without the pressure of a human presence.
  • the clinician may look for specific types of responses and emotions linked to certain topics in the questionnaire.
  • triage a triage questionnaire can accelerate and reduce staffing requirements and give clinicians almost direct evaluation with individuals from video answers that can be kept in the medical file.
  • the above-described system may also be used to send a direct emergency video message to a clinician.
  • the above-described method and system may be used to query information from local people in order to plan a trip.

Abstract

There is described a method for creating an interactive video portrait of a subject, comprising: receiving a structured questionnaire for the subject, the structured questionnaire comprising interview questions organized according to a given structure; transmitting the interview questions to the subject; receiving subject videos from the subject, each one to the subject videos comprising an answer to a respective interview question; analyzing and indexing the subject videos to generate the interactive video portrait of the subject; and storing the interactive video portrait.

Description

METHOD AND SYSTEM FOR CREATING AND SHARING INTERACTIVE VIDEO PORTRAITS
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority of US Provisional Patent Application having serial number 61/624,543, which was filed by Applicant on April 16, 2012 and is entitled "System and method for individual querying via interactive video portrait", the specification of which is hereby incorporated by reference.
TECHNICAL FIELD
[0002] The present invention relates to the field of video portraits, and more particularly to interactive video portraits.
BACKGROUND OF THE ART
[0003] For thousands of years, human beings have sought to leave traces of their existence via paintings, photographs, videos, etc. While video is commonly used, no tools are readily available to automate the process of building an interactive portrait of an individual. [0004] After the death of a cherished person, all that remains for the family and next generations is a series of photographs. In some cases, videos are also available but they are unrelated snapshots which cannot create a traditional biography.
[0005] Video biographies are usually produced by a team of interviewers with the help of a camera man. Specialized techniques and equipment (lights, lens, sequence editing, storyboard, direction, etc.) are usually required for recording the biography videos. In some instances, the biography videos are vulnerable due to the support on which they are stored and they need to be manually uploaded to a web server in order to be shared with relatives. Furthermore, the biography videos are expensive to produce since they involve professional workers. [0006] In some instances, the biography videos do not reflect the particular needs of a person since they are produced using a predefined structure for the storyboard. Typically, a video interview is linear and the time pace media comes with many constraints. One of the major constraints is the length of the video itself which usually means watching long stretches of non-interactive data. Accessing and identifying relevant information may be difficult.
[0007] Some software tools allow to keep track of lifetime stories. Lifetime stories are gathered in a random way and are therefore difficult to be retrieved. For example, Facebook™ could be useful to store time-stamped videos about a person but information stored thereon can be trivial and would not be of help to answer questions about the person from future generations.
[0008] Therefore, there is a need for an improved method and system for creating and sharing an interactive video portrait of a subject.
SUMMARY
[0009] In accordance with a first broad aspect, there is described a method for creating an interactive video portrait of a subject, comprising: receiving a structured questionnaire for the subject, the structured questionnaire comprising interview questions organized according to a given structure; transmitting the interview questions to the subject; receiving subject videos from the subject, each one to the subject videos comprising an answer to a respective interview question; analyzing and indexing the subject videos to generate the interactive video portrait of the subject; and storing the interactive video portrait.
[0010] In one embodiment, the step of transmitting the interview questions comprises transmitting the interview questions in a video format.
[0011] In one embodiment, the method further comprises a step of creating an interview video for each one of the interview questions. [0012] In one embodiment, the step of analyzing the subject videos comprises applying a semantic analysis to the subject videos and determining index keywords using the result of the semantic analysis, the subject videos being indexed using the determined index keywords.
[0013] In one embodiment, the method further comprises a step of applying a video analysis to the subject videos and determining additional keywords using a result of the video analysis, the subject videos being indexed using the index keywords and the additional keywords.
[0014] In accordance with another broad aspect, there is described a system for creating an interactive video portrait of a subject, comprising a processing unit coupled to a storing unit, the processing unit being configured to execute the steps of the above described method.
[0015] In accordance with a second broad aspect, there is described a method for providing an answer to a question of a user about a subject via an interactive video portrait of the subject, comprising: providing the user with an access to the interactive video portrait of the subject, the interactive video portrait comprising a plurality of videos answers of the subject being organized according to a given structure and each having index keywords associated thereto; receiving the question from the user; retrieving at least one of the plurality of videos that corresponds to an answer to the question of the user; and sending the retrieved video to the user.
[0016] In one embodiment, the step of receiving the question comprises receiving a user video containing the question.
[0017] In one embodiment, the retrieving step comprises applying a semantic analysis to the question and determining search keywords using a result of the semantic analysis.
[0018] In one embodiment, the retrieving step further comprises identifying at least one of the video answers having at least one keyword associated thereto that relates to at least one of the determined search keywords. [0019] In accordance with still another broad aspect, there is described a system for providing an answer to a question of a user about a subject via an interactive video portrait of the subject, the system comprising a processing unit coupled to a storing unit, the processing unit being configured to execute the steps of the above-described method. [0020] In accordance with a further broad aspect, there is described a system for creating an interactive video portrait of a subject, comprising: a questionnaire unit adapted to receive a structured questionnaire for the subject, the structured questionnaire comprising interview questions organized according to a givens structure; a subject communication unit adapted to transmit the interview questions to the subject and receive subject videos from the subject, each one to the subject videos comprising an answer to a respective one of the interview questions; and an indexing unit adapted to analyze and index the subject videos to generate the interactive video portrait of the subject, and store the interactive video portrait.
[0021] In one embodiment, the subject communication unit is adapted to transmit the interview questions in a video format. [0022] In one embodiment, the system further comprises an interview unit adapted to generate an interview video for each one of the interview questions;
[0023] In one embodiment, the indexing unit is adapted to apply a semantic analysis to the subject videos and determine index keywords using the result of the semantic analysis, the subject videos being indexed using the determined index keywords. [0024] In one embodiment, the indexing unit is further adapted to apply a video analysis to the subject videos and determine additional keywords using a result of the video analysis, the subject videos being indexed using the index keywords and the additional keywords.
[0025] In accordance with still a further broad aspect, there is described a system for providing an answer to a question of a user about a subject via an interactive video portrait of the subject, comprising: a providing unit for providing the user with an access to the interactive video portrait of the subject, the interactive video portrait comprising a plurality of videos of the user being organized according to a given structure and each having index keywords associated thereto; a retrieving unit for receiving the question from the user and retrieving at least one of the plurality of videos that corresponds or relates to an answer to the question of the user; and a transmission unit for sending the retrieved video to the user.
[0026] In one embodiment, the retrieving unit is adapted to receive a user video containing the question.
[0027] In one embodiment, the retrieving unit is adapted to apply a semantic analysis to the question and determine search keywords using a result of the semantic analysis.
[0028] In one embodiment, the retrieving unit is further adapted to identify at least one of the video answers having at least one keyword associated thereto that relates to at least one of the determined search keywords.
BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The following detailed description is to be understood in combination with the appended drawings, in which:
[0030] FIG. 1 is a flow chart of a method for creating an interactive video portrait, in accordance with an embodiment;
[0031] FIG. 2 is a flow chart of a method for retrieving videos contained in the interactive video portrait of FIG. 1 ;
[0032] FIG. 3 is a block diagram of a system for creating an interactive video portrait and providing users with access to the interactive video portrait, in accordance with an embodiment;
[0033] FIG. 4 is a block diagram illustrating an exemplary architecture for the system of FIG. 3; and
[0034] FIG. 5 illustrates an exemplary interface provided to a subject for recording subject videos. [0035] It will be noted that throughout the appended drawings, like features are identified by like reference numerals.
DETAILED DESCRIPTION
[0036] There is described a method and system for creating an interactive video portrait of a subject. The interactive video portrait comprises a set of videos of the subject in which he tells stories and/or answers questions on various topics of interest. The interactive video portrait is stored on a web platform and users such as relatives or members of a group of interest may access the interactive video portrait and optionally ask further questions. The users may query on various topics of interest via the interactive video portrait interface. [0037] The present method and system automates the process of collecting, preserving and sharing the interactive video portraits of subjects. In one embodiment, a personalized structured questionnaire is created for each subject and the structure questionnaire is presented to the subject through interview videos. The present method and system automates the video interview using speech recognition and advanced video tracking and detection tools. The present method and system "listen" for answers from the subject and automatically save, preserve and analyze the subject video responses. Users can query in natural spoken language or type text input and watch the subject video responses playing back from the interactive video portrait. The interactive video portrait simulates natural interaction that occurs during a discussion between the subject and the user. [0038] FIG. 1 illustrates one embodiment of a method 100 for creating an interactive video portrait for a subject. At step 102, basic information about the subject is collected. In one embodiment, the basic information collected at step 102 comprises information necessary for creating a subject profile. Questions such as binary questions are presented to the subject who provides an answer for each question. Examples of questions comprise questions about the subject's age, the subject's marital status, whether the subject has children, the subject's areas of interest, the subject's profession, etc.
S9] In one embodiment, the subject is guided in the process of creating a genealogy at step 102. For example, the subject may be presented with an empty genealogy tree and may complete the empty genealogy tree by providing information about his parents, sisters, brothers, children, uncles, cousins, and/or the like. The information may comprise the first name, the last name, contact information such as an email address for example, and/or the like for each contact person included in the genealogy tree. [0040] In one embodiment, a subject profile is generated for the subject at step 102. In addition to the information collected at step 102, the subject profile may further comprise subject authentication information, contact information, pictures of the subject, and/or the like.
[0041] It should be understood that the information collected at step 102 may be stored in the subject profile. [0042] In one embodiment, the subject profile may further comprise access information that identifies users who may have access to the subject interactive video portrait, and the parts of the interactive video portrait that an authorized user may access. In another example, access information may comprise an identification for members of a group of interest, users, etc. In one example, members of a group or users may be associated with a time delay so that permissions can be granted at a specific date.
[0043] At step 104, a structured questionnaire is created for the subject based on the personal information collected at step 102. The structured questionnaire is a sequence of interview questions designed to obtain statistically useful information about given topics for the subject. The content and sequence order of the interview questions are determined using the user profile and preference in order to collect the desired information about the subject. For example, if method 100 is used for collecting information about the life of the subject, then the structured questionnaire may take on the style of a typical biographer questionnaire and may use a structure that systematically covers each area of the subject's biography. The information collected at step 102 may be used for determining a priority in the flow of interview questions to be presented to the subject.
[0044] In one embodiment, the structured questionnaire is designed for collecting information about the life of a subject. In this case, the structured questionnaire includes interview questions that may be organized in themes such as family, activities, vacations, sports, etc.
[0045] In one embodiment, a respective structured questionnaire is created for each subject so that the structured questionnaire may vary from one subject to another. In this case, each structured questionnaire is specific to a given subject and the subject- specific structured questionnaire is created using the subject information collected at step 102.
[0046] In another embodiment, a single structured questionnaire is created once at step 104 and the same structured questionnaire applies for all of the subjects. In this case, the subjects are then presented with the same structured questionnaire and only respond to questions that are relevant to them. For example, if the subject does not have children, he may skip the interview questions about children.
[0047] In a further embodiment, a single generic structured questionnaire is created once at step 104 and the method 100 comprises a further step of adapting or personalizing the generic structured questionnaire for each subject. In this case, the generic structured questionnaire is personalized for each subject using the subject personal information collected at step 102. In one embodiment, the personalization of the generic structured questionnaire may be performed by generating a subject-specific path connecting at least some of the questions included in the generic structured questionnaire. For each subject, the path identifies the interview questions included in the generic structured questionnaire to be asked to the subject and/or defines the order in which the identified questions should be presented to the subject. For example, if the subject has specified that he is childless and never got married, then the subject-specific path does not connect interview questions related to spouses and children so that these interview questions are disabled and will not be presented to the subject.
[0048] An expert may create the interview questions on a topic of interest and create interrelation paths between the interview questions so as to structure the interview questions. The paths between the interview questions may regroup the interview questions per topics or themes, indicate the sequence that should be followed when asking the interview questions, indicate a temporal order for the interview questions, and/or the like. [0049] For example, if the genealogy information about the subject shows that the subject has no children, the structured questionnaire may not include interview questions about children. Alternatively, the structured questionnaire may comprise interview questions about children that would be different from the interview questions that would be asked to a subject who has children. For example, a childless subject may be asked to give the reasons why that is so while a subject having children may be asked to discuss about the preferred activities of his children.
[0050] In one embodiment, the structured questionnaire may comprise different versions of a same question and the version of the question that is suitable for the subject is selected using the information collected at step 102. For example, the suitable version of a given question may be selected as a function of the age of the subject. Furthermore, the expert may assign at least one index keyword to each interview question contained in the structured questionnaire. The index keywords allow to retrieve the answers according to the questions asked as opposed to retrieving those indexed by semantic analysis. [0051] In one embodiment, the structured questionnaire is organized as a tree where the branches each represent themes such as family, friends, work, etc., and the leaves each represent interview questions.
[0052] In one embodiment, the structured questionnaire is provided to the subject for validation. The subject may then deactivate or skip interview questions included in the structured questionnaire. The subject may also assign access information to given interview answers included in his interactive video portrait in order to prevent some users from accessing the subject videos corresponding to the given interview questions.
[0053] In one embodiment, the expert may further create multiple structured questionnaire that may be used by corporations, marketing agencies, focus group researchers, interest group managers, clinician report personnel, etc.
[0054] Referring back to FIG. 1, an interview video is created at step 106 for each interview question included in the structured questionnaire that has been created at step 104. [0055] In one embodiment, for each interview question included in the structured questionnaire, a person, such as an actor for example, records an interview video in which he asks the corresponding interview question.
[0056] In another embodiment, the interview videos are automatically generated by appropriate software/hardware. In this case, the interview videos each contain a computer- generated avatar who asks a respective responding interview questions.
[0057] The subject may be presented with a choice of actors/avatars and may select a desired actor/avatar for the creation of the interview videos.
[0058] It should be understood that the interview videos are linked together according to the same structure as that of the structured questionnaire.
[0059] At step 108, the interview videos are presented to the subject who records videos of himself, for example subject videos, while answering the interview questions. As a result, a subject video is created for each interview question at step 110. It should be understood that the subject may decide not to answer all of the questions contained in the structured questionnaire or may record more than one subject video for a same interview question.
[0060] In one embodiment, the interview videos are automatically played back according to the structure of the structured questionnaire. In this case, after the subject answers a first question contained in a first interview video, a second interview question video is automatically played back. After the subject answers the second question contained in the second interview video, a third interview video is automatically played back, etc.
[0061] In another embodiment, the structured questionnaire is provided to the subject who may modify the order in which he would like to answer the interview questions. As described above, the subject may also skip some questions and/or add additional questions. In the latter case, an interview video is created for the additional interview questions. [0062] In one embodiment, the subject may assign at least one keyword for at least some of the subject videos. The subject may also assign localization information such as an address or a location name to a subject video in order to permit subsequent identification on a map. The subject may also assign contact persons for at least some of the subject videos in order to permit or prevent identified contact persons from accessing the given videos. The contact persons that have been authorized to have access to the given subject videos may be subsequently notified that the given videos are available.
[0063] In one embodiment, the subject may further attach media content to the subject videos. For example, pictures, text, additional videos, and/or the like may be attached to a given subject video. Media content may help illustrate the answer to an interview question given in a subject video. [0064] In one embodiment, a plurality of interview videos are displayed to the subject for a same interview question. For example, a first interview video in which an actor or an avatar asks a question is first presented to the subject, and a second interview video is presented to the subject while he is answering the question asked in the first interview video. The second interview video may present the actor or avatar in an idle state in which he listens to the answer given by the subject. In this case, the second interview video is referred to as an idle video.
[0065] In one embodiment, the subject video is streamed in substantially real-time to the server while being recorded, and automatically analyzed to determine which idle video is to be presented to the subject. For example, audio and video analysis may be applied to the subject video to determine the emotional state of the subject while answering the interview question. For example, if smiles or laughs are detected in the subject video, then an idle video in which the actor/avatar smiles may be presented to the subject. In another example, if it is determined that the subject is sad while answering the interview question, then an idle video in which the actor/avatar appears to be compassionately listening is presented. In a further example, if hesitations, surprises, grimaces, etc. are detected, then an idle video in which the actor/avatar has a neutral attitude may be presented to the subject. [0066] It should be understood that the server may have stored thereon idle videos each presenting the actor/avatar in a different idle state, and the appropriate idle video to be presented to the subject is determined via video and/or audio analysis of the subject video.
[0067] In an embodiment in which the subject video is streamed in substantially real-time to the server, video analysis may be used to determine an answer to an interview question. For example, an interview video in which an actor/avatar asks whether the subject would like to answer questions about politics may be presented to the subject. The subject may answer the question by shaking his head. In this case, video analysis is performed to extract the movement of the subject's head. If it is determined that the subject shakes his head from left to right, then it is determined that the subject does not want to discuss about politics and the subject is presented with interview videos related to other topics. If it is determined that the subject shakes his head from top to bottom, then it is determined that the subject accepts to discuss about politics and the subject is presented with interview videos related to politics.
[0068] As will be readily understood, such non-verbal communication may be affected by the culture and the environment in which the subject was raised and/or lived for a substantial amount of time. For example, a subject of Indian descent may use a different head shake, such as a head bobble, to indicate an acceptance of the question or an encouragement for the actor/avatar to continue with the process. If data is available about the subject's culture, the video analysis may use this data. If data is unavailable and/or video analysis does not allow a definitive decision on the meaning of the non-verbal gesture, a video may be presented to the subject in which the actor/avatar requests a specific verbal answer.
[0069] In one embodiment in which the subject video is streamed in substantially realtime to the server, video analysis of the streamed subject video may be performed for different purposes. For example, it may be determined that the subject is no longer in the field of view of the camera, the person in front of the camera is not the subject (using face recognition methods), the subject is not moving, the eyes of the subject are closed (and the subject may be sleeping), etc. [0070] In one embodiment, the length of the subject video is determined and compared to an expected video length, such as an average video length, or a range of video lengths. The subject may be informed about whether the subject video has a "normal" length, for example a length that substantially corresponds to the expected length or is comprised within the range of video lengths. The subject may also be informed whether the subject video is shorter or longer than the expected video length. In an embodiment, in which the subject video is streamed in substantially real-time to the server, the subject may be informed in substantially real-time when the length of the subject video he is recording exceeds the "normal" length or the range of video lengths. [0071] While in the above description, the interview questions are presented to the subject via interview videos, it should be understood that the interview questions may be presented in a written form. In this case, step 106 is omitted and step 108 comprises a step of presenting to the subject the interview questions in a written form, which are displayed on the subject display unit. Similarly, the interview questions may be presented in an audio-only media. In this case, step 106 includes the steps of recording oral interview questions and storing the audio files containing the oral interview questions, and step 108 includes the steps of transmitting the audio files to the subject and playing back the audio files on the subject machine to present the oral interview questions to the subject. It should be understood that combinations of methods for presenting the interview questions to the subject are possible. For example, some interview questions may be presented to the subject in a written form while other interview questions may be presented via interview videos.
[0072] Then, at step 112, the subject videos are first analyzed and then indexed and organized according to index keywords. Natural speech recognition is applied to the audio track of each subject video in order to translate the spoken words contained in the audio track into text. It should be understood that any adequate method of natural speech recognition may be used. For example, Dragon NaturallySpeaking™ software may be used.
[0073] Then, semantic analysis is run over the text corresponding to the audio track of the subject video to generate a list of index keywords. The list of index keywords comprises at least one theme/topic that corresponds to the subject video. Optionally, video analysis may also be run over the video tracks to generate a list of facial feature keywords. For example, 30 to 60 feature facial points may be tracked to extract statistics information and interpret facial expressions. Software such as FaceAPI™ or FaceReader™ may be used for the video analysis. For each subject video, a list of corresponding keywords is generated using the keywords determined by the semantic analysis, the keywords assigned to the corresponding interview questions, and, if any, the keywords assigned to the interview video by the subject and the keywords determined from the video analysis. The determined keywords are assigned to the respective subject video. All of the subject videos form an interactive video portrait for the subject. [0074] In one embodiment, the semantic analysis is performed by a recognition engine based on finite-state transducers. The speech recognition engine applies a Large Vocabulary Continuous Speech recognition model that receives the audio track as input and outputs a lattice of assumptions, a language model for Large Vocabulary Continuous Speech having a conversational style, a pronunciation model, an acoustic model such as a Hidden Markov model (HMM) or Linear Discriminant Analysis (LDA)/ Likehood Linear Transform (LLT)/ Feature Maximum Likehood Linear Regression (fMLLR) transform matrices, etc. One example of adequate speech recognition engine comprises the Kaldi™ speech recognition toolkit.
[0075] In one embodiment, the whole written text corresponding to the audio track of the subject video is also stored and assigned to the subject video. In the same or another embodiment, the natural speech recognition may not be able to recognize and convert into text all of the words or expressions contained in the audio track of the subject video. In this case, the audio wave, for example the phonetic signature, corresponding to the word or expression that has not been recognized during the natural speech recognition step is stored in memory and assigned to the subject video. The phonetic signature may be extracted using the Kaldi™ speech recognition toolkit for example.
[0076] In one embodiment, the subject may interact with the system, e.g. he may answers to questions saying "yes" or "no". In this case, the speech recognition engine may be further adapted to detect vocal activity from the subject, reduce noise and echo, and use specific speech recognition grammar that may include rejection mechanisms.
[0077] In one embodiment, the keywords assigned to a subject interview are categorized. The keywords assigned to interview questions and the keywords assigned by the subject form a list of first-level priority keywords while the keywords generated by the semantic analysis are included in a second-level priority list. This allows users to find answers about specific words/topic in non-expected videos. For example, if a user queries the word "fish", the system will prioritize the detected "fish" words and related semantic themes such as "sports", "trout", etc. [0078] In one embodiment, for each subject video, the list of determined index keywords is provided to the subject who may add additional keywords. For example, the subject may be provided with a list of selectable keywords generated by semantic analysis based on the detected keywords from the text of the subject video, and the subject may manually deactivate unsuitable keywords which otherwise would be assigned to the respective subject video along with the automatically generated index keywords previously assigned to the subject video.
[0079] In an embodiment in which media content is assigned to a given subject video, index keywords may also be assigned to the content media. For example, a picture assigned to a given subject video by the subject may have assigned thereto the same index keywords as those that have been generated for the corresponding subject video. [0080] In one embodiment, using the natural speech recognition, names of contacts cited by the subject in a given subject video are detected. In this case, au automatic notification may be sent to the identified contact for informing the contact that the given subject video is available for play back. In another example, a confirmation request may be provided to the subject for permitting the cited contact to have access to the given subject video. The subject may confirm or not the permission. If permitted by the subject, a notification may also be sent to the contact. If no contact information is available for a detected contact name, then the subject receives a notification to enter contact information such as an email address in order to send an invitation to the detected contact. In one embodiment, the emotional state of the subject is also determined for each subject video. An expression recognition analysis is performed on the video and/or audio tracks of the subject video in order to determine the emotional state of the subject while recording the subject video. For example, laughs may be identified within the audio tracks, smiles may be detected from the video tracks in order to determine whether the subject is happy or sad. Basic statistics such as agitation, head position, mouth and eye status, etc. can also be retrieved from the video analysis. The determined emotional state may then be tagged to the corresponding subject video. In the same or another embodiment, the emotional state may correspond to an index keyword assigned to the subject video. In a health care embodiment, the system can be used to detect patterns in emotional state.
[0081] In an embodiment in which the present method 10 is used for medical purposes, the subject video may be analyzed to perform a non-verbal evaluation. Elements such as head agitation frequency, body agitation frequency, eye movement frequency, eye direction statistics, voice pitch variation, speech speed variation, skin color variation, emotion recognition, hesitation statistics, and/or the like may be determined from the video and tagged to the subject video. For example, the determined non-verbal evaluation elements may be stored in metadata assigned to the subject video.
[0082] In one embodiment, the semantic analysis performed on the subject videos is added to the subject profile. [0083] In one embodiment, the structured questionnaire is updated using the results of the semantic analysis. For example, new areas of interest for the subject may be detected using the results of the semantic analysis and questions related to the new areas of interest may be inserted in the structured questionnaire of the subject. In an embodiment in which a generic structured questionnaire is personalized for each subject, interview questions that were previously disabled may be included in the subject-specific path. In this case, interview videos may be generated for the newly inserted interview questions and subsequently presented to the subject. The structure of the structured questionnaire may also be modified according to the results of the semantic analysis. For example, if specific interests in sport are detected in the semantic analysis of previous answers, then the Sports topic section of the questionnaire will be raised in the priority list of the questions presented to the subject.
[0084] Referring back to FIG. 1, the indexed subject videos forming an interactive video portrait for the subject are stored in memory at step 114, for example each subject video and its assigned index keywords are stored in the memory.
[0085] The interactive video portrait of a given subject may then be accessed by users who may play back the subject videos corresponding to the given subject. The structured questionnaire of the given subject may be provided to the users, for example the list of the interview questions contained in the structured questionnaire may be displayed to the users who may select a given interview question and then access the subject video corresponding to the given interview question.
[0086] The interactive video portrait may also allow a virtual live conversation between a given subject and a user. In this case, the user connects to a system on which the interactive video portrait of the given subject is stored and asks a specific question. The system retrieves at least one subject video that corresponds to an answer to the specific question asked by the user, and the retrieved subject video is played back to the user. FIG. 2 illustrates one embodiment of a method 120 that allows such a virtual live conversation between a user and a given subject.
[0087] At step 122, a specific user question about a given subject is received from a user. If there is no access restriction associated with the interactive video portrait of the given subject, then any user may access the interactive video portrait. Alternatively, only the users that have been authorized by the given subject may access the interactive video portrait.
[0088] In one embodiment, the method comprises a step of providing the user with a list of questions that may be virtually asked to the given subject. In this case, the user selects a desired question from the list. The questions included in the list may correspond to interview questions that have been asked to the given subject while creating his interactive video portrait. It should be understood that the list of selectable questions provided to the user may comprise only the interview questions that were answered in the structured questionnaire by the given subject.
[0089] In the same or another embodiment, the user may ask his own question. The user question may be a written question entered by the user via an input device such as a keyboard. In another example, the user question may be asked orally via a microphone. In a further embodiment, the user may create a video containing a video and an audio tracks, in which he asks a specific question to the given subject. In this case, natural speech recognition is used for converting the audio track of the user video into text. In a further embodiment, the user may enter a text query to ask a specific question to the given subject. In still another embodiment, the user may use a combination of text, audio and video segments to formulate his question to the given subject.
[0090] After receiving the user question from the user at step 122, at least one subject video is identified from the interactive video portrait of the given subject at step 124. The identified subject videos correspond to potential answers from the given subject to the user question received from the user.
[0091] In order to retrieve the subject videos that correspond to the received user question, search keywords are first determined from the received question using semantic analysis, as described above with respect to subject videos. Search keywords such as search semantic themes/topics are extracted from the semantic analysis. The semantic themes and the detected search keywords are then extrapolated with synonym groups. The indexation gathered from the questions and the analyzed answers is used to find related themes derived from synonyms keywords and the determined related themes are ranked by relation frequency. For example, a user may ask a question about sailing. In this case, keywords related to "sailing" such as sail, regatta, water sport, wind, boat, and the like, are first determined. Then the system will first retrieve and play subject videos that are related to the sport section of the structured questionnaire, and then retrieve all subject videos having keywords corresponding to the determined related keywords whether these subject videos are contained in the sport section of the structured questionnaire or not. In an embodiment in which the user question is an oral question contained in an audio file, a step of speech recognition is first performed in order to convert the oral question into text from which search keywords can be determined. Similarly, if the received user question is an oral question contained in an audio track of a video, a step of speech recognition is first performed in order to convert the oral question into text from which search keywords can be determined. [0092] Once they have been determined from the received user question, the search keywords relevant to the index keywords and themes assigned to the subject videos, and the subject videos having at least one index keyword that corresponds to at least one search keyword may be identified as a potential answer to the received user question. The identified subject videos are then provided to the user at step 124.
[0093] In an embodiment, the natural speech recognition may not be able to recognize and convert into text all of the words or expressions contained in the audio track of the user question video. In this case, the audio wave, for example the phonetic signature, corresponding to the word or expression that has not been recognized during the natural speech recognition step is compared to the phonetic signatures assigned to the subject videos in order to find potential matches.
[0094] In one embodiment, the subject videos provided to the user are automatically played back on the subject machine. In another embodiment, the list of retrieved subject videos is provided to the user and an indication of their relevance may also be provided. In a further embodiment, only the most relevant subject video is automatically played back on the user machine and the other retrieved subject videos may be played back upon selection by the user. In this case, the user experiences a substantially live virtual conversation with the subject. Each time the user asks a question, a subject video in which the subject provided an answer to the question is automatically played back on the user machine.
[0095] In one embodiment, all of the subject videos having at least one index keyword that matches/relates to one of the search keywords are considered as potential answers and are provided to the user. In another embodiment, only the subject videos having at least two keywords that match search keywords are considered as potential answers and are provided to the user. [0096] In one embodiment, the subject videos that are provided to the user are ranked as a function of their relevance relative to the user question. The ranking may be calculated based on the number of matches between search keywords index keywords and their relative theme group, the occurrence frequency of the search keywords, and/or the like. [0097] In one embodiment, the determined search keywords are provided to the user before the determination of the subject videos that correspond to answers to the user question. The user may update the list of search keywords by adding/removing search keywords and the identification of the suitable subject videos is then performed based on the updated search keywords. [0098] In an embodiment in which an emotional state indication is assigned to the subject videos, the emotional state indication may also be provided to the user at step 124 along with the determined subject videos. Similarly, in an embodiment in which media content is assigned to subject videos, the media content is also provided to the user at step 124 along with the determined subject videos. [0099] In an embodiment, users may assign an appreciation indication to subject videos. In this case, the appreciation indication may be provided to the user along with the respective subject video. In one embodiment, users can mark a given subject video as inappropriate if content found therein is offending. In this case, the given subject video may be put in quarantine until a decision from an administrator is made. [00100] In one embodiment, users may also send comments and/or a testimony after play back of a given subject video, and a notification may be sent to the subject.
[00101] It should be understood that the subject videos are organized according to the same structure as that of the structured questionnaire. If the subject videos are organized as a tree, a user may select a specific branch of the subject video tree representing a specific topic or theme in order to specify a search criteria. In this case, the user is provided with the subject videos related to the selected branch. [00102] In one embodiment, the user may found answers of a given subject by directly browsing the structured subject videos for the given subject.
[00103] In an embodiment, a user may ask an additional interview question to the subject. In this case, the user creates an interview video in which he asks the additional interview question. The subject is then notified that an additional interview video has been posted by the user.
[00104] FIG. 3 illustrates one embodiment of a system 130 adapted to execute the methods 110 and 120. The system comprises a server 132, a subject machine 134, an expert machine 136, and a user machine 138, which are each provided with at least a processing unit coupled to a storing unit for storing data thereon and a communication unit. The subject machine 134, the expert machine 136, and the user machine 138 are connected to the server 132 via a communication network 140.
[00105] The server 132 is adapted to receive subject information from the subject machine 134 that is further provided with an input device such as a keyboard, a sound speaker, a microphone and a video camera. The server 132 is adapted to transmit the structured questionnaire from the expert machine 136 which is provided with a display and an input device such as a keyboard. An expert can generate a structured questionnaire containing interview questions as described above using the expert machine 136. The expert machine 136 transmits the structured questionnaire to the server 132. [00106] In one embodiment, the server 132 is adapted to transmit the interview questions in a written form to the subject machine 134. The interview questions are then displayed on the display unit of the subject machine 134.
[00107] In another embodiment, the server 132 is configured for generating interview videos based on the received structured questionnaire. Each generated interview video comprises an avatar asking a corresponding interview question.
[00108] In another embodiment, the system 130 further comprises an actor machine 142 provided with at least a processing unit coupled to a storing unit, a communication unit, a display unit, a video camera such as a webcam for example, and a microphone. In this case, the structured questionnaire is sent to the actor machine from the server 132 and an actor records an interview video for each one of the interview questions contained in the structured questionnaire. The interview videos are then sent to the server 132. [00109] The server 132 is further configured for indexing the received interview videos and transmit the indexed interview videos to the subject machine 134, as described above. The server 132 is also configured for receiving a user question from the user machine 138 that is provided with an input device, a sound speaker, a display unit, and optionally a microphone and/or a video camera. The server 132 is adapted to retrieve subject videos corresponding to the user questions, as described above. The server 132 then sends the retrieved subject videos to the user machine 138.
[00110] It should be understood that the interview questions may also be presented to the subject in an oral-only form. In this case, the server 132 may be adapted to automatically generate oral questions that are stored in audio files. Alternatively, the oral questions may be recorded by an actor via an actor machine 142 and then uploaded to the server 132. The server 132 then transmits the audio files to the subject machine 134.
[00111] In one embodiment, the user machine 138 is configured for automatically play back at least one of the received subject videos, as described above.
[00112] FIG. 4 illustrates one exemplary architecture for the server 132. The server 132 comprises a subject profile unit 150, a questionnaire unit 152, an interview video unit 154, a video providing unit 156, a video indexing unit 158, a video retrieving unit 160, a notification unit 162, and a database 164.
[00113] The subject profile unit 150 is adapted to retrieve questions from the database 164, transmit the questions to the subject machine 134, and receive answers to the questions from the subject machine 134. Questions from the structured questionnaire may be sent to the subject who provides an answer for each question. Examples of questions comprise questions about his age, his genealogy, his friends, etc. [00114] In one embodiment, the subject profile unit 150 is adapted to guide the subject in the process of creating a genealogy tree. For example, the subject profile unit 150 may be adapted to transmit an empty genealogy tree and the subject may complete the empty genealogy tree by providing information about his parents, sisters, brothers, children, uncles, cousins, and/or the like. The information may comprise the first name, the last name, contact information such as email address for example, and/or the like for each person included in the genealogy tree. The completed genealogy tree is then sent to the subject profile unit 150 and stored in the database.
[00115] In one embodiment, the subject profile unit 150 is further adapted to generate a subject profile for each subject. The user profile may comprise subject identification information, contact information, pictures of the subject, and/or the like. As described above, the subject profile may further comprise access information.
[00116] In one embodiment, the subject profile unit 150 is adapted to recognize human faces in order to identify the subject using a picture or video received from the subject machine 134. Face recognition or voice recognition may also identify users permissions over specific subject videos.
[00117] The questionnaire unit 152 is adapted to retrieve the subject information from the database 164 and transmit the subject information to the expert machine 136. An expert creates a structured questionnaire which is transmitted to the questionnaire unit 152 from the expert machine 136 and stored in the database 164. As described above, the structured questionnaire contains interview questions interrelated by paths to form a structure, and index keywords for each interview question.
[00118] In one embodiment, the structured questionnaire is organized as a tree where the branches each represent topics such as family, friends, work, etc., and the leaves each represent interview questions. In this case, the questionnaire unit may be adapted to provide a tree structure to the expert machine that is used for assigning interview questions to the leaves of the tree, assigning topics to the branches of the tree, adding/removing branches and leaves, etc. [00119] In one embodiment, the questionnaire unit 152 is adapted to transmit the structured questionnaire to the subject machine 134 for validation by the subject. The subject may then delete interview questions included in the structured questionnaire and/or add additional interview questions in order to further personalize the structure questionnaire. The subject may also assign access information to given interview questions included in the structure questionnaire in order to prevent some users from accessing the subject videos corresponding to the given interview questions.
[00120] In one embodiment, the interview video unit is adapted to retrieve the structured questionnaire from the database and transmit the structured questionnaire to the actor machine 142. For each interview question included in the structured questionnaire, a person, such as an actor for example, records an interview video in which he asks the corresponding interview question. The interview videos are then sent to the interview video unit 154 and stored in the database 164. It should be understood that the interview videos are organized according to the structure of the structured questionnaire. [00121] In another embodiment, the interview video unit 154 is adapted to generate interview videos in which an avatar asks a respective interview question. The generated interview videos are stored in the database 164. In this case, the interview unit 154 may be adapted to send a list of avatar to the subject machine 134 and receive an indication of a desired avatar from the subject machine 134. Then, the interview videos are generated using the selected avatar.
[00122] The video providing unit 156 is adapted to retrieve the interview videos from the database 164 and transmit the interview videos to the subject machine 134. The subject machine is adapted to play back the interview videos and record a subject video in which the subject answers the question asked by the actor or avatar in the interview video. The subject videos are sent from the subject machine 134 to the video indexing unit 158.
[00123] In one embodiment, the video providing unit 156 is adapted to sequentially transmit the interview videos to the subject machine 134 and automatically play back the interview videos on the subject machine 134. [00124] In another embodiment, the video providing unit 156 is adapted to transmit the structured questionnaire to the subject machine 134. The subject may then select a given interview question and an indication of the selected interview question is transmitted to the video providing unit 156 which retrieves the corresponding interview video and transmits it to the subject machine 134.
[00125] In one embodiment, the subject may create an additional interview question which is sent to the interview video unit 154 so that a corresponding interview video be created.
[00126] The video indexing unit 156 is adapted to receive the subject videos from the subject machine 134 and organize and index the subject videos based on index keywords. The video indexing unit is adapted to perform natural speech recognition on the audio track of each subject video in order to translate the spoken words contained in the audio track into text, as described above. The video indexing unit is further adapted to perform semantic analysis in order to determine index keywords and themes from the text of the subject videos, the structure of the structured questionnaire, the keywords assigned to the corresponding interview questions, and, if any, the keywords assigned to the interview video by the subject, as described above. The determined keywords are assigned to the respective subject video and stored in the database 164. In one embodiment, the video indexing unit 156 may further be adapted to determine phonetic signatures and assigned the determined phonetic signatures to their respective subject videos, as described above. [00127] In one embodiment, the subject may assign at least one keyword to an interview video and the assigned keyword is transmitted to the video indexing unit 158. The subject may also assign a contact person to a subject video in order to permit or prevent the identified contact persons from accessing the given videos. The subject may also assign location information to a subject video in order to access the given videos using geolocalisation. An indication of the contact person is then transmitted to the notification unit 162 that retrieves the contact information such as the email address of the identified contact person and transmits a notification that a subject video of the subject is available. [00128] In one embodiment, the video indexing unit 156 is adapted to transmit the list of determined index keywords to the subject machine 134 so that the subject who may add/remove additional keyword. The modification to the list of index keywords are transmitted to the video indexing unit 156 that updates the list of keywords accordingly. [00129] In one embodiment, the subject may further attach media content to a subject video. For example, pictures, text, additional videos, and/or the like may be attached to a given subject video. Media content may help illustrate the answer to an interview question given in a subject video. The media content is then transmitted to the video indexing unit 158 and stored in the database 164. [00130] In an embodiment in which media content is assigned to a given subject video, index keywords may also be assigned to the content media. In this case, the index keywords are transmitted to the video indexing unit 158 along with the media content.
[00131] In one embodiment, the video indexing unit 158 is adapted to recognize names of contact persons contained in the subject videos using the natural speech recognition. The names of the contact persons is then transmitted to the notification unit which is adapted to retrieve the contact information of the identified contact persons and send them a notification that a subject interview is available. Before sending the notification, the notification unit 162 may request a confirmation from the subject.
[00132] In one embodiment, the video indexing unit 158 is adapted to recognize names of places like city names for example contained in the subject videos using the natural speech recognition. The names of the places are then attached to the corresponding subject videos.
[00133] In one embodiment, the video indexing unit 158 is adapted to determine the emotional state of the subject for each subject video and the determined emotional state is assigned to the corresponding subject video. [00134] In an embodiment in which the system 130 is used for medical purposes, the video indexing unit 158 is adapted to perform a non-verbal evaluation of each subject video. The video indexing unit 158 is then adapted to determine elements such as head agitation frequency, body agitation frequency, eye movement frequency, eye direction statistics, voice pitch variation, speech speed variation, skin color variation, emotion recognition, and/or the like may be determined from the video and tagged to the subject video. These elements are then assigned to their respective subject video and stored in the database 164. [00135] The video retrieving unit 160 is adapted to provide a user access to the interactive video portrait of a subject. In one embodiment, the video retrieving unit 160 is adapted to verify user identification information received from a user machine 138 before allowing the user to access the interactive video portrait.
[00136] The video retrieving unit 160 is adapted to receive a user question about a given subject from the user machine 138 and determine the subject videos that correspond to an answer to the received user question.
[00137] In one embodiment, the video retrieving unit 160 is adapted to generate a list of questions that may be virtually asked to the given subject and transmit the generated list to the user machine 138. In this case, the user selects a desired question from the list and an indication of the selected question is sent to the video retrieving unit 160 which retrieves the subject video that contains an answer to the selected question.
[00138] In the same or another embodiment, the video retrieving unit 160 is adapted to receive a question created by the user. The received user question may be a written question entered by the user via an input device such as a keyboard. In another example, the received user question may be asked orally via a microphone. In a further embodiment, the received question may be contained in the audio track of a received video.
[00139] The video retrieving unit 160 is adapted to first determine search keywords from the received user question using semantic analysis. In an embodiment in which the user question is an oral question contained in an audio file, the video retrieving unit 160 is further adapted to convert the oral question into text from which search keywords and themes can be determined. [00140] The video retrieving unit 160 is further adapted to compare the search keywords to the index keywords assigned to the subject videos. The subject videos having at least one index keyword that corresponds to at least one search keyword or a theme may be identified as potential answers to the received user question. The video retrieving unit 160 then retrieves the identified subject videos and transmit them to the user machine 138.
[00141] In one embodiment, the user machine 138 is adapted to automatically play back the subject videos received from the video retrieving unit 160.
[00142] In another embodiment, the video retrieving unit 160 transmits the list of he retrieved subject videos. The video retrieving unit 160 may also be adapted to determine the relevance of each subject video contained in the list and transmit the determined relevance along with the list.
[00143] In a further embodiment, the user machine 138 is adapted to automatically play back only the most relevant subject video while the other retrieved subject videos may be played back upon selection by the user. In this case, the user experiences a substantially live virtual conversation with the subject. Each time the user asks a question, a subject video in which the subject provided an answer to the question is automatically played back on the user machine.
[00144] In one embodiment, the video retrieving unit 160 is adapted to transmit to the user machine 138 all of the subject videos having at least one index keyword that matches one of the search keywords or a theme group are considered as potential answers and are provided to the user. In another embodiment, only the subject videos having at least two keywords that match search keywords or theme groups are considered as potential answers and are provided to the user, etc.
[00145] In one embodiment, the video retrieving unit 160 is adapted to generate a ranking of the subject videos as a function of their relevance relative to the user question. The ranking may be performed based on the number of match between search keywords and index keywords, the occurrence frequency of the search keywords, and/or the like. [00146] In an embodiment in which an emotional state indication is assigned to the subject videos, the video retrieving unit 160 is adapted to retrieve the emotional state assigned to the subject videos from the database and transmit to the user machine 138 an indication of the detected emotional state along with the determined subject videos. Similarly, in an embodiment in which media content is assigned to subject videos, the video retrieving unit 160 is adapted to retrieve the media content assigned to the determined subject videos and transmit it along with the determined subject videos.
[00147] In one embodiment, a user may ask an additional interview question to the subject. In this case, the user creates an interview video in which he asks the additional interview question, and the user interview video is sent to the interview video unit 154 and stored in the database 164. The subject is then notified that an additional interview video has been posted by the user.
[00148] In one embodiment, a user may add an additional testimony to the subject video. In this case, the user records a testimony video in which he tells the additional interview information, and the user testimony video is uploaded to the interview video unit 154 and stored in the database 164. The subject is then notified that an additional testimony video has been posted by a user.
[00149] FIG. 5 illustrates an exemplary interface 170 that may be provided to the subject by the server 132, for example, for recording the subject videos.
[00150] The interface 170 includes a first region or area 172 for displaying the interview questions of the structured questionnaire, a second region 174 for displaying the interview video, a third region 176 for displaying the subject video while it is being recorded, a play back toolbar 178, and a recording toolbar 180.
[00151] The subject may select a given interview question from the list of questions displayed in the first region 172 and the corresponding interview video is played back in the second region. [00152] The play back toolbar 178 is used by the subject to control the play back of the interview video while the recording toolbar 180 is used by the subject for controlling the recording of the subject video.
[00153] In one embodiment, the above-described method and system may be used in a family context to collect, preserve and share intergenerational family memory. It can be used by distant relatives and friends using a rights management system preset. Intergenerational family memory can be collected, preserved and shared across multiple generations.
[00154] In one embodiment, the above-described method and system may be used to collect, preserve and share oral tradition and collective memory. Natives and others can collect, preserve and share oral cultural tradition. Oral societies will be able to keep and preserve their cultural heritage using their oral communication mode.
[00155] In one embodiment, the above-described method and system may be used in the medical field, for health problem detection, for patient pre-diagnosis, for pharmacological tracking, for therapeutic tracking, distance video diagnosis, therapy outcome measures, etc. [00156] For example, the above-described system may be adapted to generate and display graphics and charts from video detection tools vectors (emotion state, agitation level, eye move stat, voice pitch, skin color evolution, etc.) so that a clinician can get a quick state overview on a specific question or on the overall interview.
[00157] In one embodiment, the above-described system may generate and display forms and quiz aggregation in the form of charts for example.
[00158] In one embodiment, the above-described method and system may be used in managing focus groups to gather large quantities of answers, for market analysis, for marketing agencies or to collect and manage large social studies databases. As part of discussion groups (focus groups) and surveys, consumers' opinions can be collected, sorted and analyzed quickly. Questionnaires can be transmitted to large numbers of people over the internet or directly in retail stores. Similarly, marketing agencies may collect consumers' opinions. In one embodiment, the above-described method and system may be used to generate data mining request for market analysis.
[00159] In one embodiment, the above-described method and system may be used in the eLearning industry for knowledge transfer, education and fundamental research. [00160] In one embodiment, the above-described method and system may be used in the tourism industry by collecting and sharing user experiences with other users in oral mode.
[00161] In one embodiment, the above-described method and system may be used to share local information using geological digital maps found on the Internet.
[00162] In one embodiment, the above-described method and system may be used to share local information using Ground Positioning System (GPS) based tools such as mobile phones.
[00163] In one embodiment, the above-described method and system may be used for preparation and screening of candidates in the context of job interviews.
[00164] In one embodiment, the above-described method and system may be used in human resources, for rapid candidate screening, employee tracking and performance review. The system can be used in human resources to facilitate the evaluation of a large number of candidates.
[00165] In one embodiment, the above-described method and system may be used to transmit specialized knowledge of a worker to his replacement.
[00166] In the medical context, the above-described method and system may be used for example in psychiatry or grading. In psychiatry, an individual can respond much more easily, and at his own pace without the pressure of a human presence. The clinician may look for specific types of responses and emotions linked to certain topics in the questionnaire. In the case of triage, a triage questionnaire can accelerate and reduce staffing requirements and give clinicians almost direct evaluation with individuals from video answers that can be kept in the medical file. [00167] The above-described system may also be used to send a direct emergency video message to a clinician.
[00168] In the area of knowledge transfer, a researcher can produce an interactive database on his research from a predetermined questionnaire built by the participating doctoral students and colleagues working in research. Subsequently, students in related spheres may ask questions in natural spoken language to an interactive portrait of the scholar and activate corresponding response from the video files.
[00169] In the tourism field, the above-described method and system may be used to query information from local people in order to plan a trip.
[00170] Users can query information from specific regions by choosing and asking questions to portraits selected from internet geological maps.
[00171] Users can query information from local area by choosing and asking questions to portraits selected from GPS based systems such as electronic tablets and smart phones.
[00172] The embodiments of the invention described above are intended to be exemplary only. The scope of the invention is therefore intended to be limited solely by the scope of the appended claims.

Claims

I/WE CLAIM :
1. A method for creating an interactive video portrait of a subject, comprising:
receiving a structured questionnaire for the subject, the structured questionnaire comprising interview questions organized according to a given structure;
transmitting the interview questions to the subject;
receiving subject videos from the subject, each one to the subject videos comprising an answer to a respective interview question;
analyzing and indexing the subject videos to generate the interactive video portrait of the subject; and
storing the interactive video portrait.
2. The method of claim 1, wherein said transmitting the interview questions comprises transmitting the interview questions in a video format.
3. The method of claim 2, further comprising creating an interview video for each one of the interview questions.
4. The method of claim 1, wherein said analyzing the subject videos comprises applying a semantic analysis to the subject videos and determining index keywords using the result of the semantic analysis, the subject videos being indexed using the determined index keywords.
5. The method of claim 4, further comprising applying a video analysis to the subject videos and determining additional keywords using a result of the video analysis, the subject videos being indexed using the index keywords and the additional keywords.
6. A method for providing an answer to a question of a user about a subject via an interactive video portrait of the subject, comprising:
providing the user with an access to the interactive video portrait of the subject, the interactive video portrait comprising a plurality of videos answers of the subject being organized according to a given structure and each having index keywords associated thereto;
receiving the question from the user; retrieving at least one of the plurality of videos that corresponds to an answer to the question of the user; and
sending the retrieved video to the user.
7. The method of claim 6, wherein said receiving the question comprises receiving a user video containing the question.
8. The method of claim 7, wherein said retrieving comprises applying a semantic analysis to the question and determining search keywords using a result of the semantic analysis.
9. The method of claim 8, wherein said retrieving further comprises identifying at least one of the video answers having at least one keyword associated thereto that relates to at least one of the determined search keywords.
10. A system for creating an interactive video portrait of a subject, comprising a processing unit coupled to a storing unit, the processing unit being configured to execute the steps of the method of claim 1.
11. A system for providing an answer to a question of a user about a subject via an interactive video portrait of the subject, the system comprising a processing unit coupled to a storing unit, the processing unit being configured to execute the steps of the method of claim 6.
12. A system for creating an interactive video portrait of a subject, comprising:
a questionnaire unit adapted to receive a structured questionnaire for the subject, the structured questionnaire comprising interview questions organized according to a givens structure;
a subject communication unit adapted to transmit the interview questions to the subject and receive subject videos from the subject, each one to the subject videos comprising an answer to a respective one of the interview questions; and
an indexing unit adapted to analyze and index the subject videos to generate the interactive video portrait of the subject, and store the interactive video portrait.
13. The system of claim 12, wherein said subject communication unit is adapted to transmit the interview questions in a video format.
14. The system of claim 13, further comprising an interview unit adapted to generate an interview video for each one of the interview questions;
15. The system of claim 12, wherein the indexing unit is adapted apply at a semantic analysis to the subject videos and determine index keywords using the result of the semantic analysis, the subject videos being indexed using the determined index keywords.
16. The system of claim 15, wherein the indexing unit is further adapted to apply a video analysis to the subject videos and determine additional keywords using a result of the video analysis, the subject videos being indexed using the index keywords and the additional keywords.
17. A system for providing an answer to a question of a user about a subject via an interactive video portrait of the subject, comprising:
a providing unit for providing the user with an access to the interactive video portrait of the subject, the interactive video portrait comprising a plurality of videos of the user being organized according to a given structure and each having index keywords associated thereto;
a retrieving unit for receiving the question from the user and retrieving at least one of the plurality of videos that corresponds to an answer to the question of the user; and a transmission unit for sending the retrieved video to the user.
18. The system of claim 17, wherein the retrieving unit is adapted to receive a user video containing the question.
19. The system of claim 18, wherein the retrieving unit is adapted to apply a semantic analysis to the question and determine search keywords using a result of the semantic analysis.
20. The system of claim 19, wherein the retrieving unit is further adapted to identify at least one of the video answers having at least one keyword associated thereto that relates to at least one of the determined search keywords.
PCT/IB2012/054590 2012-04-16 2012-09-05 Method and system for creating and sharing interactive video portraits WO2013156828A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201261624543P 2012-04-16 2012-04-16
US61/624,543 2012-04-16

Publications (1)

Publication Number Publication Date
WO2013156828A1 true WO2013156828A1 (en) 2013-10-24

Family

ID=49383002

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2012/054590 WO2013156828A1 (en) 2012-04-16 2012-09-05 Method and system for creating and sharing interactive video portraits

Country Status (1)

Country Link
WO (1) WO2013156828A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9269374B1 (en) 2014-10-27 2016-02-23 Mattersight Corporation Predictive video analytics system and methods
CN106658167A (en) * 2016-12-13 2017-05-10 天脉聚源(北京)传媒科技有限公司 Video interaction method and device
US10621535B1 (en) 2015-04-24 2020-04-14 Mark Lawrence Method and apparatus to onboard resources

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6360234B2 (en) * 1997-08-14 2002-03-19 Virage, Inc. Video cataloger system with synchronized encoders
US20020169631A1 (en) * 2001-04-23 2002-11-14 Lewis David M. System and method for providing employers with virtual interviews of potential job candidates
US20080089665A1 (en) * 2006-10-16 2008-04-17 Microsoft Corporation Embedding content-based searchable indexes in multimedia files
WO2011004381A1 (en) * 2009-07-08 2011-01-13 Yogesh Chunilal Rathod An apparatus, system, and method for automated production of rule based near live sports event in the form of a video film for entertainment
US20110052155A1 (en) * 2009-09-02 2011-03-03 Justin Desmarais Methods for producing low-cost, high-quality video excerpts using an automated sequence of camera switches

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6360234B2 (en) * 1997-08-14 2002-03-19 Virage, Inc. Video cataloger system with synchronized encoders
US20020169631A1 (en) * 2001-04-23 2002-11-14 Lewis David M. System and method for providing employers with virtual interviews of potential job candidates
US20080089665A1 (en) * 2006-10-16 2008-04-17 Microsoft Corporation Embedding content-based searchable indexes in multimedia files
WO2011004381A1 (en) * 2009-07-08 2011-01-13 Yogesh Chunilal Rathod An apparatus, system, and method for automated production of rule based near live sports event in the form of a video film for entertainment
US20110052155A1 (en) * 2009-09-02 2011-03-03 Justin Desmarais Methods for producing low-cost, high-quality video excerpts using an automated sequence of camera switches

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9269374B1 (en) 2014-10-27 2016-02-23 Mattersight Corporation Predictive video analytics system and methods
US9437215B2 (en) 2014-10-27 2016-09-06 Mattersight Corporation Predictive video analytics system and methods
US10262195B2 (en) 2014-10-27 2019-04-16 Mattersight Corporation Predictive and responsive video analytics system and methods
US10621535B1 (en) 2015-04-24 2020-04-14 Mark Lawrence Method and apparatus to onboard resources
CN106658167A (en) * 2016-12-13 2017-05-10 天脉聚源(北京)传媒科技有限公司 Video interaction method and device
CN106658167B (en) * 2016-12-13 2020-03-17 天脉聚源(北京)传媒科技有限公司 Video interaction method and device

Similar Documents

Publication Publication Date Title
Cederström et al. Discourse of the real kind: A post-foundational approach to organizational discourse analysis
US10902841B2 (en) Personalized custom synthetic speech
US10607188B2 (en) Systems and methods for assessing structured interview responses
Jones et al. Unmanageable inequalities: sexism in the film industry
Lacey et al. Qualitative data analysis
CN111033494A (en) Computing architecture for multiple search robots and behavioral robots, and related devices and methods
Woermann Focusing ethnography: theory and recommendations for effectively combining video and ethnographic research
US20150356512A1 (en) System and Method for Optimizing Job Candidate Referrals and Hiring
Baur et al. eXplainable cooperative machine learning with NOVA
Roy The birth of a word
CN112364234B (en) Automatic grouping system for online discussion
White W et al. Laying the Foundation
Wilks Is a Companion a distinctive kind of relationship with a machine?
US20190295199A1 (en) Intelligent legal simulator
JP6088886B2 (en) Event preparation promotion advice system and method
Natali Visually exploring social perceptions of environmental harm in global urban contexts
US11423071B1 (en) Candidate data ranking method using previously selected candidate data
Wadsworth et al. Pay, power, and health: HRI and the agricultural conundrum
Pessanha et al. A computational look at oral history archives
US20170316807A1 (en) Systems and methods for creating whiteboard animation videos
US20220197931A1 (en) Method Of Automating And Creating Challenges, Calls To Action, Interviews, And Questions
WO2013156828A1 (en) Method and system for creating and sharing interactive video portraits
Wielocha et al. Collecting archives of objects and stories: On the lives and futures of contemporary art at the museum
Balamurugan et al. Impact of Artificial Intelligence on Organizational Transformation
WO2022180854A1 (en) Video session evaluation terminal, video session evaluation system, and video session evaluation program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12874460

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12874460

Country of ref document: EP

Kind code of ref document: A1