NAME

normalise_and_count.pl - normalises tokens, disambiguate lemmas, and counts occurrences.

SYNOPSIS

normalise_and_count.pl [-h|--help --man -l|--locale LOCALE] -m|--map MAP DATA

Input

DATA is a tab-delimited file with class (e.g. genre), token, lemma, and part-of-speech. Could be redirected from STDIN.

Required options

-m|--map MAP

where MAP is the file name of a mapping from token part-of-speech to lemma part-of-speech. Each row is in the following format:

 C<UNIQUE_TOKEN_POS = LEMMA_POS>

Line comments starting with # (hash sign) and empty lines are supported.

Optional options

-h|--help: prints a help message to STDOUT.
--man: prints full documentation to STDOUT.
-l|--locale LOCALE: sets the corpus-specific locale, where LOCALE is a supported locale (default=system default locale, example: sv_SE.ISO-8859-1)

Output

A tab-delimited file with token frequency, token, part-of-speech, lemma, lemma frequency and class (e.g. genre). Frequency is within-class frequency.

REQUIREMENTS

Requires Perl 5.004 or later (for locale handling).

DESCRIPTION

This Perl script normalises tokens, disambiguates lemmas based on part-of-speech, and counts occurrences for lemmas and tokens within a class (e.g. genre). Normalisation (casefolding) is done in 3 steps, and assumes normalised lemmas:

casefold if token exist in normalised form elsewhere in corpus
casefold against lemma, character by character
casefold the rest if changed in step 2

For proper normalisation, you should give the locale of your corpus as argument, or the default locale of your system is used. The locale is used for casefolding.

Lemma (or rather baseform) disambiguation is done by concatenating a lemma part-of-speech (PoS) tag to the baseform from the corpus. Lemma PoS tags are conflations from the token PoS tags. You should use a corpus-specific PoS conflation mapping which maps token PoS tags to lemma PoS tags. The map is a text file, where each row is in the following format:

UNIQUE_TOKEN_POS = LEMMA_POS

An example from the PAROLE tag set for Swedish (mapping a plural, definitive, genitive form of a neuter common noun to a neuter common noun, where neuter is a lexical feature):

NCNPG@DS = NCN

The file format supports line comments starting with # (hash sign) and empty lines.

Prints to STDOUT.

LICENSE

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.

AVAILABILITY

http://stp.lingfil.uu.se/~evafo/resources/basevocpool/

VERSION HISTORY

Created: 2005-01-19: Eva Forsbom

$Log: normalise_and_count.html,v $

Revision 1.1 2006/08/11 12:31:59 eva

HTML documentation for Perl scripts

Revision 1.3 2006/08/11 12:10:04 eva Added pod usage to Perl files

Revision 1.2 2006/08/08 12:14:12 eva Added pod documentation.

2005-09-09 Eva Forsbom * moved corpus-specific corrections to correct_<CORPUS>.pl * moved locale specification to argument * moved corpus-specific PoS-conflating to property file (argument)