Configure Near-Duplicate Detection in IPRO Analytics

Near-Duplicate Detection, by definition, is the identification of groups of documents in which the majority of the text is the same and in the same order as the other items in the group, but with some minor additions or removals.

Click here to watch the Near Duplicates video

After the configuring and processing a near-duplicate set, in Eclipse, you can:

• Near Duplicate Relationship Reference - View related near duplicates in the Relationship panel within Eclipse during review. This can also be used in conjunction with the two other methods. For more information, see Set Up the Relationship Panel to Display Near-Duplicates (Analytics).

• Near Duplicate Batch Review - Set up batch review by near duplicate group to allow for documents in the same ND group to be reviewed by one reviewer all at the same time, providing for faster, more consistent review decisions. For more information, see Create Near Duplicate Batches for Review (Analytics).

• Near Duplicate and Email Thread Batch Review - Use Email Threads to review emails and their attachments, and Near Duplicate technology to review the remaining loose documents. For more information, see Create Email Thread with Near Duplicate Batches for Review (Analytics).

Near-Duplicate Detection Configuration Workflow

The following diagram outlines the process for configuring and processing a near-duplicate detection set.

Note: When the near-duplicate set is processed, the metadata of the items identified as part of a near-duplicate family will be updated with a group ID, which is used to identify the near-duplicate group to which the items belong.

Near-duplicate Fields

Fields used for near-duplicates analysis are listed in the following table. These fields are created at the time you configure near-duplicates analytics. Fields are required unless otherwise noted.

Field Created in ipro eclipse	description
IA ND Family	A group identifier to identify all items in the same textual near duplicate group. This value will be the item ID of the principal item in the group, which is the item against which all other group members were compared with, and which has the most useful text of any item in the group. .
IA ND Score	The similarity of this item to the principal item of a near-duplicate group. This value is expressed as a decimal percent in the range of [0-100]. The principal item of the group and any exact duplicates of it will always have a value of 100. No item will have a value less than the Minimum Similarity setting (expressed as a percentage) used when near-duplicate detection was performed.
IA ND Sort	A value for items that can be used to sort near-duplicates in a group so that the principal document is first and all other group members follow in order from the highest to the lowest similarity. If multiple groups need to be sorted, the groups should first be sorted by their near dupe family value and then be sorted by this value.
IA ND Words	The number of duplicative words in the document compared to the principal item of the near-duplicate group.

Configure Near-duplicate Detection

Preparation:
1. Review details in Near-duplicate Fields to understand these fields and determine whether any existing fields in the case represent comparable data. If they do, make a note of the existing fields.
2. Evaluate case data and documents for spreadsheets and numeric data; determine whether or not these should be included in the near-duplicates analysis.
3. Determine the needed similarity cutoff, using a value from 1 to 100. The default level is 25. A higher number represents a greater minimum similarity.
Start IPRO Eclipse Administration and log in as a user who is a member of the Super Admin group.
On the Case Management ribbon, select IPRO Eclipse Analytics > IPRO Eclipse Analytics Near Duplicate Detection.
Select the needed managing client and case.
On the left side of the screen, map near-duplicates fields. Required fields are identified by True in the Required column. Either automatically or manually map needed fields as follows:
1. To automatically map fields (and create if needed), click Auto-Map Required Fields. Newly created fields are marked with an asterisk, *.
2. To manually map an Analytics Field to a field in the case, click in the Database Field cell and select the needed field from the list.
3. If you are manually mapping fields, repeat step b until all needed fields are mapped.
Near Dupe Set Name: If you are creating a new Near Dupe set, select the Create a new set option and enter a unique name to identify it.
Previously Processed Set: Alternatively, if you would like to use a previously processed set, select the Use previously processed set option and select the Near Dupe set you would like to use.
Documents to Compare: Select one of the following options:
- To analyze all documents in the case, select Entire Case.
- To analyze a subset of documents based on a search saved in IPRO Eclipse, select the search, and click OK.
Configuration: To use a previously created and saved configuration, click the button and select the configuration you would like to use.

Click here to watch the Configurations video

Manage Configurations: To set the options for the configuration profile you would like to use, click the Manage Configurations button to open up the dialogue and set the following options:

Item	Description
Name	Enter a name for the configuration profile.
Similarity Function	Select the function you would like to use for determining similarity between two documents. Literal Jaccard is the default value: Literal does NOT take document length into account when determining similarity. This means that similarity between two documents is determined only by the number of words in common. Literal Jaccard does take document length into account when determining similarity. This means that the similarity between two documents is determined by the number of words in common AND the differences in word count, typically resulting in a more accurate measurement of similarity:
Text Assumption Level	Select one of the following options to determine how documents will be treated when analyzing for non-binary documents (i.e. documents that contain readable text as opposed to just numbers): 0: Only treats documents as non-binary after the system has analyzed the text and identified the content as containing prose. 5: If the document was converted from a known format to plain text, it assumes that the result should be treated as non-binary. 10: In addition to the assumptions made at level 5, if the document is in a format that can’t be translated into text, it assumes that the document can be treated as non-binary without further translation of validation.
Similarity Cutoff	Enter the Similarity Cutoff value (from 1 to 100) to be used for determining the threshold for how similar two documents can be. The default value is 25.

Click Process.
Wait as documents are analyzed. The amount of time required depends on the number of documents being analyzed and the near-duplicates configuration. When the analysis is complete, a success message displays.
Implement near-duplicates detection as appropriate to your site. For example, you might have reviewers refer to the near-duplicates relationship. See Manage Relationships for details. Or, you might create and save a search for a near-duplicates field.
Alert your users to the availability of near-duplicates and how this feature should be used at your site.

Create Near Duplicate Batches for Review (Analytics)

Create Email Thread with Near Duplicate Batches for Review (Analytics)