| Excerpt |
|---|
In this example, you create a text labeling project to analyze the sentiment in a Tweet, and use Majority Vote for Labeling QA. |
Project Overview
You want to create a project that enables you to label the sentiment in a given Tweet post as Positive, Negative, Neutral, or Not Sure.
To create this project, you must perform the following tasks:
This document explains how you can perform each of the tasks listed above. Specific sections in this document also contain sample data that you can use to easily create and implement this project on Taskmonk.
Listing Project Requirements
In this project, you want to:
Identify the sentiment type in a given Tweet as Positive, Negative, or Neutral.
In the cases where the sentiment cannot be determined by you, mark it as Not Sure.
Sample Input Data
You select sample tweets that you wish to annotate. You paste the text of the tweets into a Microsoft Excel sheet under a column labeled Tweet. You save the Microsoft Excel sheet as Tweet_Tagging_Input.xlsx.
Download Source File
You can follow the steps listed above to create your Microsoft Excel Sheet or download this file, unzip it, and use it in your project: Tweet_tagging_input.xlsx.
Configuring Project Metadata
Project Metadata is the first tab that appears when you create a project. The Project Metadata tab enables you to provide basic information, such as the name, process, and project type, associated with your project. You can also upload any documentation that you may want to add to your project.
For detailed information on working with the Basic Info section of the Project Metadata tab, see Add Basic Project Information.
For detailed information on working with the Basic Info section of the Project Metadata tab, see Manage Project Documentation.
To create the project, click the Create Project floating button to the left side of the Projects page.
The Project Metadata tab associated with your new project appears.
2. Enter Twitter Post Analysis as the Project Name and Sentiment Identification as the Process.
3. Click Project Type > Text-Based.
4. Click Next.
The Documents sub-tab appears.
5. You can upload documents associated with the project if required. This is an optional step, and you can skip it for now.
Click Next.
The Task Design tab appears. Use this tab to manage your project input and output fields.
Managing Project Input and Output Fields
Project input and output fields are key elements that determine what happens in your project. The input fields that you specify here will appear as available options for input in your project. Similarly, the output fields that you configure here will appear as output options in your project execution UI. In other words, your project can only uptake and output data associated with the input and output fields that you create here.
Taskmonk uses the project type that you specify to add input and/or output fields to projects as required. You can modify these later. In this instance, you selected Text-Based as the Project Type, and select the field type for the input and output fields you’ve created under Task Design.
Click the Input Field tab to display the Input Field UI.
Click the Create Input Field button.
To edit the input field details:
Click the Edit Field Name icon and enter Tweet in the Name textbox.
Click the Update Field Name icon next to the Name textbox to save the changes.
Click the Field Type dropdown and set it to Text.
4. In this example, we’ve created two input fields, Tweet#and Tweet. Set Field Type to text, Mandatory to True, and Operational to True for both these fields.
Creating Output Field Details
Click the Output Field tab to display the Output Field UI.
2. Click the CREATE OUTPUT FIELD button.
3. In the Create Output Field modal, set the values for the following:
Name: Provide the output field name.
Set Data Type: Select from the listed options how the output field options will appear. In this case, select Radio.
Possible Values: Type in comma-separated values that this output field can take. These values will be listed for radio button selection.
4. Select the All Levels check box to indicate that this field must be available to users at all execution levels.
5. Click Create to save the new output field.
6. Click Next to move to the next step, and click the Quality Workflow tab.
Creating Quality Workflows
The Quality Workflow tab enables you to specify how you want to ensure output quality. It also helps you create the execution levels required for your quality workflows. For example, you want to create the following levels for this project:
Annotator
Quality Analyst
Delivery Lead
In this instance, you want to enable a Majority-Vote workflow. Click the Execution Method field and select Majority-Vote from the drop-down list that appears.
2. By default, Taskmonk creates the Analyst role for you. Click the Update Execution Level icon. The Edit Execution Level modal appears.
3. Enter Annotator as the Execution Level Name.
4. You can skip all other fields here. Click Update to return to the Quality Workflow tab.
5. Click the Add Execution Level button on the right side of the page. The Add Execution Level modal appears.
6. Enter Quality Analyst in the Execution Level Name field.
7. You can skip all other fields here. Click Add.
Taskmonk adds the new execution level, closes the modal, and displays the updated Quality Workflow. Repeat the process to add all required execution levels.
8. Click Next to update User counts and Majority vote.
a. Set the User counts to 3 if every task has to be allocated to 3 different users. Majority vote 2 indicates that out of 3 users, if at least 2 of them provide the same response, the task completes at level 1. If all 3 users provide different responses, the task moves to level 2.
b. Consensus Percentage: Percentage of total tasks imported that should be allocated to multiple users. The remaining tasks will follow the Maker-Checker or Maker-Editor model.
Managing Users and Roles
You must now add users to your project and assign the execution levels you just created to them.
Click the Users tab just above Quality Workflow to view the Manage Users section.
Click the Add button on the top-right.
3. In the Add User window, select the users from the list for various levels (L1, L2, and L3).
The minimum number of users in level 1 should be equal to the User counts set in the workflow.
4. Click Add to add the selected users to their respective execution levels.
The users get added successfully to the Manage Users page.
Managing Project Datasets
Your project is now configured. Congratulations!
Before you can start labeling, you must upload the input files containing the raw data to be labeled.
Click the Datasets tab. The Datasets page appears. Use this page to manage datasets for your project.
2. Taskmonk organizes datasets into batches to simplify management and tracking. To add a new dataset, click Add Batch on the right side of the page. The Add Batch modal appears.
3. Enter Batch 1 as the name for the batch that you want to import in the Add New Batch field. You can ignore the other fields.
4. Click Submit. This creates a new batch of data for your project and adds it to the Pending tab of the Datasets page.
5. You can now upload datasets into the batch, as required. To add a dataset to the batch, click the Import button under the Tasks(Import/Export) column. The Import Task modal appears.
6. Click Choose Files, select the sample input file from your computer and click Import.
7. Once the dataset is imported, click Close to exit the modal.
8. All the tasks get updated under the Pending section.
Labeling Text Using Taskmonk
Log in as Analyst and click the My Tasks icon at the top of the page to view the My Tasks page.
2. Click the Get Tasks button adjacent to the Twitter Post Analysis project. The labeling UI associated with this project appears.
3. You can see the following project details in the labeling UI:
Batch (Batch 1) at the top-left section of the page.
Input fields (Text) at the top half of the page.
Output/Result fields at the bottom half of the page.
4. Select the option for Sentiment Type and click Submit & Get Next Task.
5. Repeat the step until all tasks are completed
For detailed information on working with a typical labeling UI, see Labeling Data.
Viewing Batch Status Report
Go to the Projects page and click Reports > View for the Twitter Post Analysis project. The Reports page appears.
Click Dataset Progress Reports to view the batch status report. This shows the total number of tasks pending and completed at each level for all batches. The Disagreement score indicates the total number of tasks without a majority at level 1. In other words, the total tasks which moved to level 2.
Downloadable Sample Files
| Attachments | ||
|---|---|---|
|