$Id: README,v 1.3 2009/05/10 10:44:56 evafo Exp $

README for BaseVocabulary package
---------------------------------

Contents:
---------

Description
License
Files
Requirements
Installation
Example runs
Documentation
Version history


Description:
------------

This package includes a Swedish base vocabulary, based on the
Stockholm-Ume Corpus, and an English base vocabulary, based on the
Susanne corpus, and scripts for creating the base vocabularies and
computing various frequency and dispersion measures. The package is
licensed under the GNU General Public License. It is available from
http://stp.lingfil.uu.se/~evafo/resources/basevocpool/.

More details on the base vocabularies can be found here: Forsbom,
Eva. 2006. Deriving a base vocabulary pool from the Stockholm-Ume
Corpus. Term paper for NGSLT course Soft Computing
(http://stp.lingfil.uu.se/~evafo/gslt/dataanalysis/datanalysis06forsbom.pdf).

The scripts were written solely for the purpose of the paper, and have
been tested only for Linux (2.4.22, Mandrake 9.2 and 2.6.14, Fedora
Core 4). The base vocabularies, however, are raw text files, and can
be viewed in any editor.


License:
--------
Copyright (C) 2005-2006 Eva Forsbom
                                                                                
Eva Forsbom, Uppsala University/GSLT                                            
E-mail: evafo@stp.lingfil.uu.se                                                 
URL: http://stp.lingfil.uu.se/~evafo/resources/basevocpool/                  
Address:                                                                        
Dept. of Linguistics and Philology                                              
Box 635                                                                         
SE-751 26 Uppsala                                                               
SWEDEN                                                                          
                                                                                
This program is free software; you can redistribute it and/or                   
modify it under the terms of the GNU General Public License                     
as published by the Free Software Foundation; either version 2                  
of the License, or (at your option) any later version.                          
                                                                                
This program is distributed in the hope that it will be useful,                 
but WITHOUT ANY WARRANTY; without even the implied warranty of                  
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the                   
GNU General Public License for more details.                                    
                                                                                
You should have received a copy of the GNU General Public License               
along with this program; if not, write to the Free Software                     
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.



Files:
------

README  - this file
bin     - directory for scripts and binaries
data    - directory for vocabularies
doc     - directory for documentation
gpl.txt - GNU General Public License

bin/

    adjusted_frequency.pl - Perl script for computing Adjusted Frequency

    basevoc_suc.sh - Shell script for deriving base vocabulary from SUC

    basevoc_susanne.sh - Shell script for deriving base vocabulary from Susanne

    conflate_suc.map - Conflation map for SUC (used by normalise_and_count.pl) 

    conflate_susanne.map - Conflation map for Susanne (used by
        normalise_and_count.pl)

    contribution.pl - Perl script for computing Contribution

    correct_suc.pl - Perl script for correcting "anomalies" in SUC annotation 

    correct_susanne.pl - Perl script for correcting "anomalies" in
        Susanne annotation

    dispersion_and_fmod.pl - Perl script for computing Dispersion and
        Modified Frequency

    merge_wordforms.pl - Perl script for merging wordform info into
        lemma base vocabulary

    normalise_and_count.pl - Perl script for normalising,
        disambiguating, and counting

    susanne2data.pl - Perl script for extracting token info from Susanne file

    xces2r.xsl - XSLT template for extracting token info from XCES file (SUC)

data/ 

    SUC_basevoc  - A Swedish base vocabulary based on Stockholm-Ume Corpus 2
    Susanne_basevoc  - An English base vocabulary based on Susanne (R5) corpus

doc/ 

    adjusted_frequency.html
    contribution.html
    correct_suc.html
    correct_susanne.html
    dispersion_and_fmod.html
    merge_wordforms.html
    normalise_and_count.html
    susanne2data.html



Requirements:
-------------

The base vocabularies are raw text files, and can be viewed by any editor.

The scripts, in Perl (5.005) and XSLT, were all developed for a Linux
environment using standard modules, but they are probably portable to
other environments (sorry, I have no way of testing), except for the
shell scripts basevoc_suc.sh and basevoc_susanne.sh, which are used as
glueing batch scripts for the other scripts. (Use them as examples
rather than as turnkey scripts.)

The SUC corpus* can be obtained, subject to a license, from
http://www.ling.su.se/dali/suc/suc2.0_info.html. The original corpus
files can be converted from SGML format to valid XML format with
parole2xml.pl (http://stp.lingfil.uu.se/~evafo/software/).

* Stockholm-Ume Corpus, version 2, 2002, Stockholm University,
Department of Linguistics and Ume University, Department of
Linguistics.

The Susanne (R5) corpus can be downloaded from
http://www.grsampson.net/RSue.html. Its annotation scheme and corpus
compilation (excerpts from the Brown corpus) are described in the
following book: Geoffrey Sampson. 1995. English for the Computer: The
SUSANNE Corpus and analytic scheme. Clarendon Press, Oxford. ISBN
0-19-824023-6.



Installation:
-------------

Unpack:
- - - -
$ tar -xzf BaseVocabulary.tgz
$ cd BaseVocabulary

(Optional)
Edit shell scripts (basevoc_CORPUS.sh) to your environment:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

(replace with your path to this package)
    BINDIR=$HOME/cvs/BaseVocabulary/bin
           ---------
(replace with your path to a temporary directory)
    TMPDATA=$TMPDIR/BVdata
            --------------

(Optional)
Edit shell scripts (basevoc_CORPUS.sh) for your corpus:
- - - - - - - - - - - - - - - - - - - - - - - - - - - -
* basevoc_susanne.sh:

(replace with the file name to your tar-gzipped Susanne corpus)
    tar -xOzf ~/corpora/SUSANNE.tgz fc2/[A-N]* |\
              ---------------------
(or, use cat if you have un unzipped copy)
    cat YOUR_PATH/fc2/[A-N]* |\
        ---------

* basevoc_suc.sh:

(replace with the path to your xmlified PAROLE SUC files)
(see e.g. parole2xml.pl at http://stp.lingfil.uu/~evafo/software)
    xsltproc $BINDIR/xces2r.xsl ~/corpora/SUC2/xml/*
                                --------------------
(if you don't have xsltproc, change to another XSLT processor)
(or, write your own extraction script for the unxmlified corpus files)



Example runs:
-------------

In general:
- - - - - -

The scripts for frequency and dispersion can be run for other material
than corpus or text data. Refer to the documentation for those
scripts.

    adjusted_frequency.{pl|html}
    contribution.{pl|html}
    dispersion_and_fmod.{pl|html}

The xces2r.xsl script might be used to extract token info from other
corpora in XCES/TEI format, given they have the same level of
annotation as SUC, and the text class i reflected in the text ID. If
xsltproc is used as XSLT parser, the following command will write data
files to the current working directory:

 $ xsltproc YOUR_PATH/BaseVocabulary/bin/xces2r.xsl XCES_FILE(S)

The other scripts are specific for base vocabulary derivation, or for
processing the SUC or Susanne corpora.


Deriving the Susanne base vocabulary:
- - - - - - - - - - - - - - - - - - -

 $ YOUR_PATH/BaseVocabulary/bin/basevoc_susanne.sh > SUSANNE_BASE_VOCABULARY


Deriving the SUC base vocabulary:
- - - - - - - - - - - - - - - - -

 $ YOUR_PATH/BaseVocabulary/bin/basevoc_suc.sh > SUC_BASE_VOCABULARY


Processing your own corpus:
- - - - - - - - - - - - - -

If you want to process another corpus, the corpus has to have
annotations for baseform and part-of-speech (PoS) for each token, and the
texts have to be classified (into genres or domains or the like).

 1. Copy basevoc_susanne.sh.

    $ cd BaseVocabulary/
    $ cp bin/basevoc_susanne.sh bin/basevoc_CORPUS.sh

 2. Enter the locale of the corpus in basevoc_CORPUS.sh. For Susanne
    it is en_US, for SUC it is sv_SE.

    # settings for normalisation
    CORPUS_LOCALE=en_US
                  -----

 3. Enter a contribution filter threshold for the corpus in
    basevoc_CORPUS.sh. For Susanne it is 2, for SUC it is 4.

    # contribution filter threshold setting
    FILTER_THRESHOLD=2
                     -

    You have to experiment with this parameter, e.g. by starting at 1
    and successively try values up to n - 1 (where n is the number of
    text classes). If possibly, you can do this in parallel with
    deriving base vocabularies for any subdivisions, and comparing the
    ranking.

 4. Write a mapping from token PoS to lemma PoS in the following
    format:

    UNIQUE_TOKEN_POS = LEMMA_POS

    The mapping is used in normalise_and_count.pl to disambiguate
    between homographic baseforms for different PoSs. The token PoS
    has to be unique, but the lemma PoS could occur more than once. An
    inflected verb in Susanne might be mapped to a verb lemma, as in
    the following example:

    VVZv = V

    And an inflected neuter noun in SUC might be mapped to a neuter
    noun, as in the following example (since neuter is a lexical
    feature, and there are cases where neuter and non-neuter paradigms
    have the same baseform):

    NCNPG@DS = NCN

 5. Enter the file name of your mapping in basevoc_CORPUS.sh. For Susanne
    it is conflate_susanne.map.

    # settings for normalisation
    CONFLATION_MAP=$BINDIR/conflate_susanne.map
                   ----------------------------


 6. Write a script for extracting corpus info into the following
    tab-delimited format:

    CLASS    TOKEN    BASEFORM    POS

    A Susanne line might look like this:

    A01a    deserves        deserve VVZv

 7. Enter a command for the script, redirecting the output to the
    temporary directory, in basevoc_CORPUS.sh. For Susanne and SUC,
    this is done under step 1 in their corresponding shell scripts.


 8. For Susanne and SUC, the text IDs reflect the classification of
    texts in that the first character represents the genre. The text
    IDs are therefore stripped to the first character under step 2 in
    the corresponding shell script. If you need to remap your corpus
    classes, replace the sed command line under step 2 in
    basevoc_CORPUS.sh. Delete it otherwise. Keep the cat command line,
    though.

    
    cat $TMPDATA/*.dat |\          # keep, but adapt file name if necessary
                ------
    sed 's/^\(.\).../\1/' |\       # optional, adapt command if necessary
                          --       # but, if adapted, remember the pipe!


 9. For Susanne and SUC, some annotations are corrected under step 3
    in their corresponding shell scripts. If you need to do any
    corrections, write a script and enter the command line for it
    under step 2 in basevoc_CORPUS.sh (as in the previous step,
    remember the pipe!). Delete the lines for step 3 otherwise.

10. Compute the base vocabulary.

    $ bin/basevoc_CORPUS.sh > data/CORPUS_basevoc



Documentation:
--------------

* HTML documentation for perl scripts:
  (in directory doc, but can be created)
  $ cat bin/adjusted_frequency.pl | pod2html > doc/adjusted_frequency.html
  $ cat bin/contribution.pl | pod2html > doc/contribution.html
  $ cat bin/correct_suc.pl | pod2html > doc/correct_suc.html
  $ cat bin/correct_susanne.pl | pod2html > doc/correct_susanne.html
  $ cat bin/dispersion_and_fmod.pl | pod2html > doc/dispersion_and_fmod.html
  $ cat bin/merge_wordforms.pl | pod2html > doc/merge_wordforms.html
  $ cat bin/normalise_and_count.pl | pod2html > doc/normalise_and_count.html
  $ cat bin/susanne2data.pl | pod2html > doc/susanne2data.html


* Online full documentation in man page format for perl scripts:
  $ bin/SCRIPTNAME.pl --man


* Online short help to STDOUT for perl scripts:
  $ bin/SCRIPTNAME.pl -h


For the other scripts and data files, any documentation can be found
directly in the files.


Version history:
----------------

Created 2006-08-04 Eva Forsbom (evafo@stp.lingfil.uu.se)

$Log: README,v $
Revision 1.3  2009/05/10 10:44:56  evafo
Updated URL to SUC.

Revision 1.2  2006/08/11 12:08:28  eva
Added info on merge_wordforms.pl.
Updated example runs.

Revision 1.1  2006/08/09 21:52:27  eva
README for BaseVocabulary package.
