UIT The arctic university of Norway > Giellatekno
 

Meeting_2005-10-31

Meeting setup

  • Date: 31.10.2005
  • Time: 10.00 Norw. time
  • Place: Wherever we are : -)
  • Tools: iChat, SubEthaEdit, phone

Agenda

  1. Opening, agenda review
  2. Reviewing the task list from two weeks ago
  3. Documentation - divvun.no
  4. Corpus gathering
  5. Corpus infrastructure
  6. Linguistics
  7. Speller infrastructure
  8. Other issues
  9. Summary, task lists
  10. Closing

1. Opening, agenda review, participants

Opened at 10: 05.

Present: Børre, Maaren, Sjur, Thomas, Tomi, Trond

Absent: Saara

Main secretary: Tomi

Agenda accepted as is.

2. Reviewing the task list from the last meeting

Børre

  • Contact oahpahusossodat and the rest of the SD about texts
    • Get help from the Tromsø department of Sámediggi to dig in WebSak
      • Done, and picked up some texts
  • Gather public texts
    • From the Sámediggi
  • Reorganise the directory structure
    • Put all corpus texts into one place
      • Done
    • Continue converting text from input format to our xml
      • Not done
  • Ask Thor-Øivind to move bugzilla to our new webserver.
    • He has been very busy, and since bugzilla seems to work ok, he has scheduled the update for next week.

Maaren

  • The missing list, both the overall missing list from our xml corpus, and a file-for-file review, in order to get different terminology.
    • Not done
  • continue working with the missing list from risten.no
    • Not done
  • Start working on Sámi place names
    • Not done
  • Start working at normativity issues (numeral issues with Trond?)
    • Not done, sorry

Saara

  • Look at the corpus infrastructure issue
  • Look at the corpus interface issue with Lars
  • make an emacs mode for the name project (cf. specs in the memo above)
    • Done
  • Plan a conversion script for the name lexicon.
    • Not done, discussed with Tomi about using his c++ code in xml2lexc script.

Sjur

  • Lule Sámi twol problems, look again at the sets definition with Thomas and Trond
    • Not done
  • risten.no bugs and fixes
    • installed the latest eXist snapshot, and tested it
    • corrected several conformity bugs that had been accepted by earlier snapshots
  • discuss risten.no work with Tomi
    • Not done
  • follow up on voice group-chat not working to Sámediggi
    • Test Marratech
  • project planning with Trond, continued
    • also look at the development processes - specification and testing
  • Follow up on place names from Norge Digitalt -> remind Bjørn Olav Megard
    • He is reminded, but no response so far. Needs more action.
  • Evaluate SFST as speller (and analyzer) lexicon
    • more thorough analysis than was possible in Guovdageaidnu
  • write a background document on the corpus contracts
    • Not done
  • Discuss the contract issue with Trond, return the new version to the lawyer
    • Call Kimmo Koskenniemi for comments
      • no answer so far
  • write to the Giellalávdegoddi once more, emphasizing timetable and response needs in the Divvun project
    • wrote draft letter and sent it to Maaren for QA and translation
  • discuss kvensk project support with Trond
    • Not done
  • write public tender documents
    • Nothing last week
  • other tasks:
    • finally corrected the Forrest config for XXE to be fully installable independent of the application itself. This will make it much easier to install upgrades of XXE, and also to upgrade the config itself, which is important to help the old GIO take control of the risten.no development.
    • The new config is available from me
    • you should all update to XXE 3.0: -)
  • buy:
    • rucksacks
    • new computer (project server)?
    • project management software
    • OmniOutline (upgrade)
    • OmniGraffle (upgrade)
    • ISDN card for Maaren (Maaren will order herself)

Thomas

  • work on Lule Sami compounding and derivation
    • finished (this calls for at least a virtual celebration !!)
  • Look at Linguistic bugs with Trond
  • Meet with Sjur and Trond about the definition of G1, G2, G3
    • not done

Tomi

  • Aspell: Continue working on the affix file & aspell
    • Contact aspell author (UTF-8 thing)
      • Not done
  • three-part compounding
    • Not done
  • corpus infrastructure: dtd location (both public and internal)
    • Not done
  • corpus infrastructure: file and dir organisation
    • Done
  • Document aspell and corpus infrastructure
    • Partially done
  • Cgi-script for uploading documents to corpus base
    • Done, but needs modifications?
  • Specification for new catxml in C++
    • this includes also placing the source and binary
      • Not done
      • clean the script/ catalogue with Trond
  • Common makefile issues
    • Not done
  • discuss risten.no work with Sjur
    • Not done

Trond

  • Work on the bug list (7 open).
    • Still 7 open, as the number project hasn't started.
    • Started looking at the G3 definition issue for sme, it is the key to some of the bugs, and responsible for very many of the missing words.
  • project planning with Sjur, continued
  • also look at the development processes - specification and testing
    • Not done.
  • Work on the name project:
    • Introduce the +Mal, +Fem, ... tags to the parser
    • Done. Now the tags are there, and in the multichar list. We may thus start writing rules for them in the sme-dis.rle file.
    • and discuss the work with Maaren and Børre.
    • Not done. Hopefully
  • clean the script/ dir
    • Down from 76 to 51 entities.
  • discuss kvensk project support with Sjur
    • Hmm, did we discuss this issue?
  • Otherwise, the week was dominated by going to a conference on minority languages and language technology, in Bozen.

3. Documentation

Documentation tasks:

  1. Add documentation on our corpus infrastructure and our corpus work in general ("To be done by the ones making the corpora": Børre, Tomi, Trond, Saara).
  2. Now we have 4 documents:
    1. Correct corpus (disamb usage)
    2. Corpus plan (for the disamb corpus cwb)
    3. catxml

For the basic corpora, we need 3 types of documentation, or doc for 3 target groups:

  1. For the users/linguists: What corpus are found, how do I use them (this info is now scattered)
  2. For the collectors: How do I add texts, where do I add them, how do I convert them (this is the Corpus conversion doc)
  3. For the programmer: What did I actually do? (this is partly the catxml doc)

For the work on the graphical user interface, we need documentation as well, in principle along the same lines, except that the user is not the same linguist as above.

  • add/update Aspell documentation (Tomi)
    • Some documentation has been written, but there still is work to be done.
  • as always: document what you're doing: -) (all)

4. Corpus gathering

Governmental documents (earlier in pdf, now in html)

Tasks:

  • move existing gov. documents (pdf) from gt/ to our corpus repository (Børre)
    • There are appr. 10 non-broken pdf documents in gt/sme/corp/original/ (the ones named stmelXXX.pdf contain only one page each)
  • Collect public (pdf and html) files (Børre)
    • Done some test downloading, will have to look at tools to do this automatically.

Contracts

Tasks:

  • Follow-up on the lawyers' comments (Trond has started with the university)
    • Trond and Sjur finished the next revision of the contracts, and are waiting for comments from Kimmo Koskenniemi
      • Update: No comments from Kimmo yet
  • add a background document explaining the model (Sjur)

The most problematic issue:

Who has the copyright of extracted material, like single words, collections of words, syntactic structure (potentially with some words filled in)? We need this to be controlled by us, not by the authors. The exact borderline is hard to define.

North Sámi New Testament

Our inhouse sme nt is as new as the one they have at Bibelselskapet, and we were told we could just use the version we have ourselves.

Lule Sámi New Testament

Svenska Bibelsällskapet is putting their finishing touches to the Lule Sámi translation, we will have it soon.

  • Haven´t heard anything from Olavi Korhonen. The only problem with our version is that there are some run together strings, and we could solve it ourselves.

Lule Sámi Dictionary

Nothing new about the meeting with Anders Kintel.

5. Corpus infrastructure

Updated task list:

  1. Make a system for file and directory permission (today: we all belong to the cvs group), to only allow people with root user privileges write access to the corpus repository, at least regarding original files
  2. Include the xsl files under version control (cvs? rcs?)
  3. Incorporate language detection as part of the corpus processing.
  4. we need a way to deal with hyphenated documents in catxml/preprocess:
    1. in normal cases hyphenation points should be removed
    2. when testing the robustness of our parsers, as well as when testing the hyphenator, the hyphenation points should be retained

6. Linguistics

Name lexicon

Summary: see the newsgroup

Unclassified:
6090 entries

3203 LONDON
1788 BERN
 468 NYSTØ
 330 ACCRA
  80 MARJA
  59 NIILLAS
  43 ANAR
  43 ALEUHTAT
  20 GIEDDI
  17 HEANDARAT
  16 DUORTNUS
   7 SULLOT
   4 VARGGAT
   4 GEAVNNIS
   3 EATNAMAT
   2 NYOBL
   2 GUOLBBA
   1 PIERA

Motivation:

  • Divvun: We want to cross-link different versions of the same locations in different languages
  • Common: We do not want to enter the same names twice. We want a language-independent name lexicon
  • Disamb: Having a richer tag set makes it easier to disambiguate
  • Future: Richer analysis makes new applications possible, within information retrieval, grammar checking, machine translation etc.

Needed: A plan for this project:

  1. do the main markup in the present propernoun file
  2. make a script for converting it to xml (to be done one time)
  3. make a script for xml2lexc (to be done by the makefile)
    1. There is a sample file for the xml file format in gt/common/src/proper-nouns.xml
    2. There is a working xml2lexc for Komi, written by Saara
  4. make the tags etc. in the parser

Conversion:

  1. Mark up the remaining 6090 entires until conversion starts (Maaren to do the Sámi names, Ilona to look at C-FI-NEN and other Finnish names, Trond, Thomas and Børre to look at the rest)
  2. Entries still to be done: see above
  3. This means we would need a seventh option, the unspecified name.
  4. Then split propernoun-sme-lex.txt into two, one with the sami name being generated by the xml2lexc script, and one manually written file, containing the name sublexica (called propernoun-sme-morph.txt or whatever)
  5. Look into efficient editing of the XML lexicon (Tomi, Saara)
  6. Then convert to xml (Tomi, Saara)
  7. Look into efficient editing of the XML lexicon again (Tomi, Saara)
  8. Look into synchronisation issues with risten.no - we want the names there as well (Tomi)
    1. Consider automatic sorting on commit

Twol SETS definition issue

The definition of G1, G2, G3 in Lule Sámi is still open. and we would like to have input on this issue. We need a G3 definition for North Sámi also.

Update: it is still not working, see bug 193

SUGGESTION (Trond): Thomas, Trond and Sjur didn't meet last week either and should try again this Wednesday instead.

North Sámi

  • three-part compounds issue still open
  • number project still open
  • The treatment of Sámi place names, we need a contract with "Norge digitalt", via UFD.
    • Sjur has written an e-mail to the UFD contact person, Øystein Johannessen, who will look into it soon. He has not responded beyond saying he will return to it. Sjur brought this up in the board meeting, and Bjørn Olav Megard will remind Øystein Johannessen about this issue. Sjur will follow up on this one.
  • normativity issues:
    • the Giellalávdegoddi meeting was last Friday, they will have a new meeting in December. They were not able to make any decisions, and there will be a new Giellalávdegoddi beginning next year who won't make decisions until late spring. This is a serious problem for the Divvun problem.
      • Actions: Sjur will bring this to the Divvun board, write a new letter to the Giellalávdegoddi, emphasizing the needs and timetables of the project
    • The document with the list of open issues needs updating, both regarding the status of each issue, and documentation of them. Also a better classification of the issues would be nice.

Lule Sámi

Sjur, Thomas and Trond will cont. Lule Sámi issues.

Numerals

  • The issue is postponed to next week.
  1. An empirical overview
    1. Numeral generation
    2. Numeral inflection
    3. Numerals as parts of compounds
  2. A clear concept of how we want to treat them
    1. Tagging
  3. A treatment

We will return to this issue after the name conversion.

7. Speller infrastructure

Nothing this week either.

8. Other

Technical issues

  • The mac os / perl bug (at least Trond and Sjur has it):
    • utf8 "\xC4" does not map to Unicode at /Users/trond/gt/script/preprocess line 82. This msg did not show up in 10.3 (perl 5.8.1), but does so in 10.4 (perl 5.8.6). It is probably a perl - OS mismatch. (Trond, Thor Øivind, Tomi)
      • Another example of the same bug:
      • : "\x{00c3}" does not map to utf8 at ../script/preprocess line 113, <> chunk 33.
      • One way to "resolve" this is to redirect the error messages to /dev/null:
... | preprocess 2> /dev/null | lookup ...

Video conferencing across firewalls

The problem we've had with the SD firewall persists, and there doesn't seem to be any resources available to help us. Geir Kaaby instead suggested we look at the Marratech package, and try it out. So please download the MacOS X client (or get it from me), and I'll send you the URL to the meeting room as soon as I get it.

Bug fixing

19 open bugs (and 24 risten.no bugs)

Bugzilla update

When Bugzilla is being moved, it should also be updated to the newest version, and the UTF-8 bug should be resolved.

Buying

  • rucksacks for the whole Divvun team

risten.no

  • Organisation: could Tomi be used, in exchange for more linguistic work by (old) GIO members? Yes, it is ok, but how much still needs to be evaluated
  • it is ok to integrate "kvensk" placenames with risten.no
    • this should be integrated with the general proper name work - we want all proper names integrated with risten.no, df above
    • needs further development of risten.no to allow for multiple XML bases to be presented and maintained in parallel. This is to be further worked on by Tomi and Sjur

Project planning and development processes

Trond is using his project as a test case for an IT guy, Geir Tore Voktor, who is taking a course in project management. Be prepared to answer questions.

Conference report from Trond

  • Relevant themes on the conference for us:
    • Terminology work
    • Dictionary work between minority languages
    • Repositories for minority language resources
    • Disambiguation work, for South African languages
  • Also, our work is relevant to other projects. This
  • There is welsh work on terminology:
    • ( Dewi Jones, Delyth Prys, U Wales, Bangor, I couldn't find). They use ISO standards for terminological work, the ones of us interested in risten.no should have a look.

9. Summary, task list

Børre

  • Contact oahpahusossodat about texts
  • Gather public texts
  • Continue converting text from input format to our xml
  • Ask Thor-Øivind to move bugzilla to our new webserver
    • ... and update Bugzilla at the same time
  • install Marratech client to Maaren's computer
  • install new XXE and the new XXE Forrest config for all (or check that it is installed and working)
  • mark-up names
  • move existing corpus docs from gt/ to new corpus repository

Maaren

  • continue working with the missing list from risten.no
  • Start working on Sámi place names
  • Start working at normativity issues (numeral issues with Trond?)

Saara

  • Look at the corpus infrastructure issue
  • Look at the corpus interface issue with Lars
  • make an emacs mode for the name project (cf. specs in the memo above)
  • look into efficient editing of the xml proper name lexicon (tools, modes, etc)
  • start looking at conversion of the name lexicon from present format to xml

Sjur

  • Lule Sámi twol problems, look again at the sets definition with Thomas and Trond, Wednesday
  • risten.no bugs and fixes
  • discuss risten.no work with Tomi
  • follow up on voice group-chat not working to Sámediggi
    • Test Marratech
      • install Marratech client to Maaren's computer
  • project planning with Trond, continued
    • also look at the development processes - specification and testing
  • Follow up on place names from Norge Digitalt
    • write an e-mail to or call Bjørn Olav Megard
  • Evaluate SFST as speller (and analyzer) lexicon
    • more thorough analysis than was possible in Guovdageaidnu
  • write a background document on the corpus contracts
  • Discuss the contract issue with Trond, return the new version to the lawyer
    • Call Kimmo Koskenniemi for comments, perhaps arrange a meeting with him
  • Follow up on meeting with Anders Kintel
  • discuss kvensk project support with Trond
  • write public tender documents
  • buy:
    • new computer (project server)?
    • project management software
    • OmniOutline (upgrade)
    • OmniGraffle (upgrade)

Thomas

  • do main markup in the present propernoun file
  • work on North sámi compounding and derivation
  • Look at Linguistic bugs with Trond
  • Meet with Sjur and Trond about the definition of G1, G2, G3

Tomi

  • Aspell: Continue working on the affix file & aspell
    • Contact aspell author (UTF-8 thing)
  • three-part compounding
  • corpus infrastructure: dtd location (both public and internal)
  • Document aspell and corpus infrastructure
  • Specification for new catxml in C++
    • this includes also placing the source and binary
      • clean the script/ catalogue with Trond
  • Common makefile issues
  • discuss risten.no work with Sjur
  • discuss about xml-processing with Saara
  • look into efficient editing of the xml proper name lexicon (tools, modes, etc)
  • start looking at conversion of the name lexicon from present format to xml
  • Look into synchronisation of proper names with risten.no

Trond

  • Work on the CG-related bugs on the bug list (7 open) (numeral related ones postponed.
  • project planning with Sjur, continued
    • also look at the development processes - specification and testing
  • Work on the name project:
    • Discuss the conversion with Maaren and Børre
    • mark up names
  • discuss kvensk project support with Sjur
  • Work on the G3 bug issue with Sjur and Thomas

10. Next meeting, closing

7.11.2005 10: 00

Closed at 11: 06