This document specifies a process for collecting a new corpus of meetings in the IDIAP Smart Meeting Room. This document is a working draft that is expected to be updated and augmented throughout the data collection process. This follows from an earlier data collection effort that resulted in a corpus of 60 scripted meetings (30 train, 30 test), each of 5 minutes duration (now available at \textsf{mmm.idiap.ch}). The current data collection effort aims to address some of the limitations of the previous corpus, as well as to cater for a richer variety of research tasks

Bengio, Samy

Gatica-Perez, Daniel

McCowan, Iain A.

This document specifies a process for collecting a new corpus of meetings in the IDIAP Smart Meeting Room. This document is a working draft that is expected to be updated and augmented throughout the data collection process. This follows from an earlier data collection effort that resulted in a corpus of 60 scripted meetings (30 train, 30 test), each of 5 minutes duration (now available at \textsf{mmm.idiap.ch}). The current data collection effort aims to address some of the limitations of the previous corpus, as well as to cater for a richer variety of research tasks.LIDIA

Infoscience - École polytechnique fédérale de Lausanne

 ICOMMUNCATIONIDIAPD a l l e M o l l e I n s t i t u t efor Perceptua l Art i f i c ia lIntelligence • P.O.Box 592 •Martigny •Valais • Switzerlandphone +41− 27− 721 77 11fax +41− 27− 721 77 12e-mail secretariat@idiap.chinternet http://www.idiap.chMeeting Data CollectionSpecificationsIain McCowan a Daniel Gatica-Perez aSamy Bengio aIDIAP–Com 03-10December 2003a IDIAPIDIAP–Com 03-10 11 OverviewThis document specifies a process for collecting a new corpus of meetings in the IDIAP Smart MeetingRoom. This document is a working draft that is expected to be updated and augmented throughoutthe data collection process.This follows from an earlier data collection effort that resulted in a corpus of 60 scripted meetings(30 train, 30 test), each of 5 minutes duration (now available at mmm.idiap.ch).The current data collection effort aims to address some of the limitations of the previous corpus,as well as to cater for a richer variety of research tasks.Some specific motivations for collecting a new corpus of meetings include the need for :• More natural data : Meeting scenarios in the previous corpus were highly constrained andunrealistic.• Statistically significant results : In order to properly train and test statistical models, moredata is required in terms of number of meetings and number of basic events happening duringa meeting.• Richer research tasks : We wish to research richer sets of events occurring in meetings,particularly making more use of the visual modality.To give natural data, the technique proposed in this document is to generate a meeting based onscenarios, rather than scripts. Scenarios are given to meeting participants (or actors) before a meetingto inform them about the context and purpose of the meeting.2 Meeting and Corpus Requirements1. In total, there should be approximately 200 meetings.2. There should be a variable number of people per meeting (3-5), although the majority shouldbe 4 person meetings.3. The meeting duration should be long enough for natural behaviour and allow for an appropriatenumber of events to occur. For this we suggest durations of 15-60 minutes.4. Some people in the database should have consistent roles across meetings.5. Recording should start before participants enter the room and end after they leave.6. Meeting recordings should continue if real-life interruptions (mobile phones, people needing toleave, technical support) occur. Similarly, if meetings diverge or finish without achieving theirgoals, this is okay.7. Meetings may be stand-alone or multi-session (e.g. projects).8. Artifacts (documents, pens, laptops, water, coffee) should be included in meetings.9. Attention should be paid to keeping a balanced corpus in terms of gender and cultural back-ground of participants.10. Some meetings should contain only native English speakers, some only non-native, and somea mixture. Possibly this may be useful also at the accent level (Australian, Indian, French,German).11. Care should be taken to include meetings with non-technical content.12. The look of the room should vary across the corpus, using pictures, notice-boards, plants, etc.2 IDIAP–Com 03-1013. Some meetings should have agendas.14. At the end of the meeting, participants should be encouraged (but not obliged) to allow copiesto be made of any notes/documents they took into the meeting, or wrote during the meeting.3 Meeting Scenarios3.1 Real Meetings1. The Meeting Recording Project (20 meetings): Those involved in recording this meetingcorpus can meet to discuss progress and take decisions on issues that arise. The meeting wouldbe informally chaired by the corpus administrator, who would be reporting on progress toresearchers. Use of an agenda is encouraged for these meetings.2. Creation of a Reading Area (≈ 5 meetings): There is interest in creating a reading area atIDIAP. Staff and students could meet to discuss requirements, location, furnishings, etc. A seriesof meetings would include initial brainstorming with different students, drafting a first proposal,presenting the proposal to management, coming up with a final proposal based on feedback, anda follow-up meeting once the room has been created. A particular person should be responsiblefor the project, and should chair the meetings. This scenario should include artifacts in theform of plans, drawings, and involve people drawing on paper or whiteboard. The presentationto management could then have these diagrams as slides, as well as a equipment lists and anestimated budget. Use of an agenda is encouraged for these meetings.3. Course Design (3-7 meetings): Seniors could meet to propose a course to be taught atthe university. A first meeting could cover initial brainstorming and discussion of the idea.Following this, participants can be asked to come back with some specific input (documents,slides) for a subsequent discussion meeting at a given date. A third meeting would then takedecisions to come up with the course outline. Another idea would be to include sub-groupmeetings : a main committee with one representative from each group (speech, vision, machinelearning, interaction) would hold the above meetings, and further meetings would be organisedfor example for the speech representative to discuss with other speech staff. These meetingsshould all be chaired.4. Conference Report (≈ 5 meetings): When people return from conferences/workshops, theyshould hold a meeting to present what they learnt, how their presentation was received, anddiscuss emerging trends. As people at IDIAP already write an email report of the conference,this could be used as a presentation slide or document to guide the discussion.5. Book Club (≈ 10 meetings): People could form a book club where they decide on a common(non-technical) book, read it in their spare time and meet to discuss it. This could happen eachweek where they discuss an agreed number of chapters, or every few weeks to discuss entirebooks. Depending on the number of interested people, multiple groups could be formed. Aneffort should be made to recruit people outside work to participate in these meetings. Thismeeting would be informal and unchaired, and involve coffee and cookies.6. Drafting Job Description (≈ 2 meetings): Staff members from the different groups willget together to discuss the needs for a new employee (e.g., a new research engineer), brainstormabout the desired profile, and to agree on a concrete job description to be posted. The typicalscenario could include either two short meetings (one to brainstorm and assign somebody tocome up with a draft, and another one to refine and finish the task), or only one longer meeting.7. Technical Reading Group (≈ 10 meetings): Staff members (students or researchers) willget together to decide on organizing a reading group on a technical theme of common interest,IDIAP–Com 03-10 3to review and discuss relevant papers on the field, every one/two weeks for X weeks. Meetingswill be informal; one person (the organizer) will be the general moderator, and one person ateach time will be the presenter. Everyone should have read the article beforehand, and bring acopy (with their notes) to the meeting. The presenter can use slides if they think appropriate.Note that, as the meetings are being recorded and distributed, participants should be advisedin advance not to make disrespectful comments regarding the author and their work. Of coursesome technical criticisms of the work are to be expected, and are acceptable.8. Presentation Rehearsals: Students or researchers will rehearse a presentation (e.g. TAM,conference dry-run) in front of three or four people. Free discussion during or after the pre-sentation. To encourage participation, presenters will be given the chance to watch themselvesoff-line to improve their presentation skills.9. Torch/Skorch Design Meetings (≈ 10 meetings): The design teams for the Torch/Skorchsoftware libraries could meet periodically to discuss problems, design issues, releases, etc. Useof an agenda is encouraged for these meetings.10. ISSCO Meetings (≈ 10-20 meetings): ISSCO have offered to visit IDIAP to record anumber of meetings. For ISSCO to specify.3.2 Artificial Scenario Meetings1. Plagiarism: As defined in appendix of ‘Group Discussion as Interactive Dialogue or as SerialMonologue’.2. Investors Creating Company: Definition of this to be obtained.For this category of meetings, other scenarios will be recorded if they are specified by researchpartners.4 AnnotationEach meeting should be annotated for1. speech recognition,2. speaker turns, and3. event languages (see list below).Speech recognition and speaker turn transcriptions should ideally be sub-contracted to a specialistcompany, or possibly temporary dictation secretaries. This should be refined and checked for quality.For the event languages (listed below), an on-line program should be used to allow the recordingsupervisor to annotate as much as possible in real-time (during meeting recording sessions). ‘TheObserver’ has been suggested as an ideal software for this purpose, as well as for subsequent off-lineannotations of the event languages.Oﬄine, a minimum of two people should annotate each meeting for each set of events using sucha software tool. For events that vary on a scale (here level of interest and mood), a numerical ratingshould be given at specified intervals, such as once every 15 seconds. For other events (here groupfocus of attention, enriched turn-taking and multi-modal dialog acts), the annotation tool could beused just to mark the boundaries between events.Once this is done, annotations should be analysed for agreement between annotators. For thescale-based events, this can be done by calculating the correlation coefficient between each pair ofannotators. For the other events, this should be done using kappa statistic. A precise methodologyto do this will be specified.4 IDIAP–Com 03-10Once the agreement between annotators has been analysed, they need to be combined in someway. For scale-based, this could be the mean of the annotators (potentially after normalising eachannotator by their mean and variance). For the others, a majority-vote rule can be used (necessitatinga minimum of three annotations in cases of disagreement between annotators).4.1 Group Event LanguagesGroup Focus-of-Attention• standing presentation.• seated presentation.• (standing) whiteboard.• seated participant doing monologue.• notes (reading or writing)• seated dictating (one speaking, others writing+silent) - rare in our meetings• standing dictating (one speaking, others writing+silent) - rare in our meetings• unfocused (none of the above)Enriched turn-taking• monologue• dialogue (discussion between two people only)• group discussion• floor (discussion controlled by one person)• group silenceMultimodal dialog acts A small set of events based on dialog modelling, but related to the group.This set of events still needs to be defined.Mood (emotion)• positive (e.g. happiness)• neutral (relaxed)• negative (e.g. anger, frustration)Level of interest• engaged (focussed)• neutral• disengaged (bored)IDIAP–Com 03-10 55 ScheduleWe propose the following basic schedule for data collection, assuming a T0 of December 2003.Commencement of Data Collection (T0): Meetings will be recorded according to this specifica-tion. This will be overseen by an administrator, responsible for the definition of actual scenarios,and managing logistics of the data collection and annotation.Review of Data Collection and Annotation Procedures (T0+2): Based on the first record-ings, the data collection and annotation specification will be reviewed and potentially revised.Completion of Data Collection (T0+6)Completion of Annotation (T0+8)

Meeting Data Collection Specifications

http://infoscience.epfl.ch/record/82977/files/com03-10.pdf

Meeting Data Collection Specifications

Abstract

Similar works

Full text

Available Versions

Infoscience - École polytechnique fédérale de Lausanne