perl.perl4lib

perl.perl4lib http://www.nntp.perl.org/group/perl.perl4lib/ ... Copyright 1998-2008 perl.org Tue, 07 Oct 2008 23:39:17 +0000 ask@perl.org RE: Regular Expression for non-Roman characters by Jacobs, Jane W Firstly thanks to Michael for solving my problem of locating non-Roman characters in Marc Texts. It works like a charm! Our ILS wonks out if it gets the new Authority Records with cross-references from non-Roman Scripts. For example: 010 n 99034155 Çz no2005062324 040 DLC Çb eng Çc DLC Çd OCoLC Çd DLC Çd OCoLC 100 1 Guo, Fucheng, Çd 1965- 400 1 Kuo, Fu-chÊ»eng, Çc singer Çw nne 400 1 Kwok, Arron, Çd 1965- 400 1 Kwok, Aaron, Çd 1965- 400 1 Kwok, Fu Shing, Çd 1965- 400 1 Gwok, Fu Sing, Çd 1965- 400 1 éå¯å, Çd 1965- Our vendor sensibly recommends that we change 4XX to 5XX. This works fine for a manual load, but in a batch load context one would anticipate problems. Hence the program. Michael has encouraged me to share my result with my result with the group, even though I have my doubts about whether any of you couldnât crank out something better off the top of your heads. However, Iâm offering it up with the caveats that it is somewhat inelegant and contains at least a few steps Iâm sure are not necessary except that I canât figure out the right way to pull them out. It is slavishly derivative of the MARC::DOC::Tutorial âUpdating subject subfield x to subfield vâ. As far as I can tell it does the job but any bugs are no doubt mine. I added a $9 to my non-Latin headings to distinguish them from other âlegitimateâ 5XX tags and that, of course, can be knocked out. That said, here it is for anyone who wants it: use strict; use MARC::Batch; my $batch = MARC::Batch->new('USMARC','VerAuthtest.mrc'); open( OUT, '>VerAuthtest.dat' ) or die $1; while ( my $record = $batch->next() ) { my $leader = $record->leader(); my $Charset = substr($leader,9,1); my $Type = substr($leader,6,1); # go through all 4XX fields in the record. foreach my $Xref ( $record->field( '4..' ) ) { # extract subfields as an array of array refs. my @subfields = $Xref->subfields(); # setup an array to store our new field. my @newSubfields = (); my $newtag = $Xref->tag(); $newtag =~ s/^4/5/; # use pop() to read the subfields backwards. while ( my $subfield = pop( @subfields ) ) { # for convenience, pull out the subfield # code and data from the array ref. my ($code,$data) = @$subfield; unshift( @newSubfields, $code, $data ); } my $Xrefstring = $Xref ->as_string( 'abcq' ); if (!($Xrefstring =~ m/\p{Latin}/) && (($Type eq 'z') && ($Charset eq 'a'))) { my $newXref = MARC::Field->new( $newtag, $Xref->indicator(1), $Xref->indicator(2), @newSubfields, 9=>'Non-Latin' ); $Xref->replace_with( $newXref ); ##my $escape = '_('; } elsif (($Xrefstring =~ m/_\(/) && (($Type eq 'z') && ($Charset eq ' '))) { my $newXref = MARC::Field->new( $newtag, $Xref->indicator(1), $Xref->indicator(2), @newSubfields, 9=>'Non-Latin' ); $Xref->replace_with( $newXref ); } } # output the potentially changed record as MARC. print OUT $record->as_usmarc(); } **Views expressed by the author do not necessarily represent those of the Queens Library.** Jane Jacobs Asst. Coord., Catalog Division Queens Borough Public Library 89-11 Merrick Blvd. Jamaica, NY 11432 tel.: (718) 990-0804 e-mail: Jane.W.Jacobs@queenslibrary.org FAX. (718) 990-8566 The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.-----Original Message----- From: Doran, Michael D [mailto:doran@uta.edu] Sent: Thursday, September 25, 2008 5:54 PM To: Jacobs, Jane W Cc: perl4lib@perl.org Subject: RE: Regular Expression for non-Roman characters Hi Jane, In a MARC-8 character set environment, I would assume that the key to detecting non-Latin characters would be the presence of an escape sequence to indicate a switch to an alternate character set (e.g. Arabic, Greek, Cyrillic, etc) [1]. Everything from that point on would be non-Latin until there was an escape sequence back to Latin. In a MARC Unicode character set environment, if you are using Perl for your regular expression matching, you can probably take advantage of the Unicode \p{} constructs [2]. Something along the lines of... \P{Latin} ..which means doesn't belong to the Latin script (lowercase 'p' = belongs to, uppercase 'P' = does not belong to). For more info on the regular expression Unicode scripts/blocks see this tutorial: http://www.regular-expressions.info/unicode.html I'll point out that when I've used Unicode \p{} constructs in a program, it was necessary to explicitly label strings as being Unicode (assuming they are, natch) before regex matching, using... decode('UTF-8',$string_tobe_matched); I know that's not exactly what you asked for, but (assuming I didn't misunderstand your question) it may suggest some approaches should you end up tackling it yourself. -- Michael [1] MARC 21 Specification > ACCESSING ALTERNATE GRAPHIC CHARACTER SETS http://www.loc.gov/marc/specifications/speccharmarc8.html#alternative [2] Perl > Unicode Regular Expression Support Level http://perldoc.perl.org/perlunicode.html#Unicode-Regular-Expression-Support-Level # Michael Doran, Systems Librarian # University of Texas at Arlington # 817-272-5326 office # 817-688-1926 mobile # doran@uta.edu # http://rocky.uta.edu/doran/ > -----Original Message----- > From: Jacobs, Jane W [mailto:Jane.W.Jacobs@queenslibrary.org] > Sent: Thursday, September 25, 2008 1:24 PM > To: perl4lib@perl.org > Subject: Regular Expression for non-Roman characters > > Hi folks, > > I'm wondering if anyone has codified a regular expression that would > indicate the presence of non-Latin characters. I want to detect the > presence of non-Roman letters in authority records. Currently > Authorities with non-Roman forms of name place these in the > 4XX fields. > Our system can't handle that so I want to flip them to 5XX > and possibly > add a subfield to note what they but first I need something to detect > them > > I had in mind something like \xE0-\xFE which detects > diacritics nicely. > I'd prefer not to figure it out for myself if someone else has already > done it. > Thanks in advance. > JJ > > **Views expressed by the author do not necessarily represent those of > the Queens Library.** > > Jane Jacobs > Asst. Coord., Catalog Division > Queens Borough Public Library > 89-11 Merrick Blvd. > Jamaica, NY 11432 > tel.: (718) 990-0804 > e-mail: Jane.W.Jacobs@queenslibrary.org > FAX. (718) 990-8566 > > > > > > The information contained in this message may be privileged > and confidential and protected from disclosure. If the reader > of this message is not the intended recipient, or an employee > or agent responsible for delivering this message to the > intended recipient, you are hereby notified that any > dissemination, distribution or copying of this communication > is strictly prohibited. If you have received this > communication in error, please notify us immediately by > replying to the message and deleting it from your computer. > http://www.nntp.perl.org/group/perl.perl4lib/2008/10/msg2671.html Thu, 02 Oct 2008 05:24:18 +0000 RE: Regular Expression for non-Roman characters by Doran, Michael D Hi Jane, In a MARC-8 character set environment, I would assume that the key to detecting non-Latin characters would be the presence of an escape sequence to indicate a switch to an alternate character set (e.g. Arabic, Greek, Cyrillic, etc) [1]. Everything from that point on would be non-Latin until there was an escape sequence back to Latin. In a MARC Unicode character set environment, if you are using Perl for your regular expression matching, you can probably take advantage of the Unicode \p{} constructs [2]. Something along the lines of... \P{Latin} ...which means doesn't belong to the Latin script (lowercase 'p' = belongs to, uppercase 'P' = does not belong to). For more info on the regular expression Unicode scripts/blocks see this tutorial: http://www.regular-expressions.info/unicode.html I'll point out that when I've used Unicode \p{} constructs in a program, it was necessary to explicitly label strings as being Unicode (assuming they are, natch) before regex matching, using... decode('UTF-8',$string_tobe_matched); I know that's not exactly what you asked for, but (assuming I didn't misunderstand your question) it may suggest some approaches should you end up tackling it yourself. -- Michael [1] MARC 21 Specification > ACCESSING ALTERNATE GRAPHIC CHARACTER SETS http://www.loc.gov/marc/specifications/speccharmarc8.html#alternative [2] Perl > Unicode Regular Expression Support Level http://perldoc.perl.org/perlunicode.html#Unicode-Regular-Expression-Support-Level # Michael Doran, Systems Librarian # University of Texas at Arlington # 817-272-5326 office # 817-688-1926 mobile # doran@uta.edu # http://rocky.uta.edu/doran/ > -----Original Message----- > From: Jacobs, Jane W [mailto:Jane.W.Jacobs@queenslibrary.org] > Sent: Thursday, September 25, 2008 1:24 PM > To: perl4lib@perl.org > Subject: Regular Expression for non-Roman characters > > Hi folks, > > I'm wondering if anyone has codified a regular expression that would > indicate the presence of non-Latin characters. I want to detect the > presence of non-Roman letters in authority records. Currently > Authorities with non-Roman forms of name place these in the > 4XX fields. > Our system can't handle that so I want to flip them to 5XX > and possibly > add a subfield to note what they but first I need something to detect > them > > I had in mind something like \xE0-\xFE which detects > diacritics nicely. > I'd prefer not to figure it out for myself if someone else has already > done it. > Thanks in advance. > JJ > > **Views expressed by the author do not necessarily represent those of > the Queens Library.** > > Jane Jacobs > Asst. Coord., Catalog Division > Queens Borough Public Library > 89-11 Merrick Blvd. > Jamaica, NY 11432 > tel.: (718) 990-0804 > e-mail: Jane.W.Jacobs@queenslibrary.org > FAX. (718) 990-8566 > > > > > > The information contained in this message may be privileged > and confidential and protected from disclosure. If the reader > of this message is not the intended recipient, or an employee > or agent responsible for delivering this message to the > intended recipient, you are hereby notified that any > dissemination, distribution or copying of this communication > is strictly prohibited. If you have received this > communication in error, please notify us immediately by > replying to the message and deleting it from your computer. > http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2670.html Thu, 25 Sep 2008 14:54:02 +0000 Regular Expression for non-Roman characters by Jacobs, Jane W Hi folks, I'm wondering if anyone has codified a regular expression that would indicate the presence of non-Latin characters. I want to detect the presence of non-Roman letters in authority records. Currently Authorities with non-Roman forms of name place these in the 4XX fields. Our system can't handle that so I want to flip them to 5XX and possibly add a subfield to note what they but first I need something to detect them I had in mind something like \xE0-\xFE which detects diacritics nicely. I'd prefer not to figure it out for myself if someone else has already done it. Thanks in advance. JJ **Views expressed by the author do not necessarily represent those of the Queens Library.** Jane Jacobs Asst. Coord., Catalog Division Queens Borough Public Library 89-11 Merrick Blvd. Jamaica, NY 11432 tel.: (718) 990-0804 e-mail: Jane.W.Jacobs@queenslibrary.org FAX. (718) 990-8566 The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2669.html Thu, 25 Sep 2008 11:27:14 +0000 Position Announcement: Evergreen Systems Developer by Bill Erickson * Apologies for cross-posting * Equinox Software, Inc. / The Evergreen Experts / http://esilibrary.com Systems Developer, Library Automation Applications Does working with a small, agile team of software designers and developers to create unique and successful solutions to interesting problems sound more like a hobby than a job? Equinox is looking for self-motivated, enthusiastic software developers to join the Equinox development team. If you're a programmer with at least 2 years of systems (non-web) development experience then you may be that person. Preferred skill set: * Experience with Perl a must, additional experience with C, Javascript, Python, and/or Java a plus. * Experience with RDBMS-backed systems software * Experience with Linux * Experience working with a development team * Experience contributing to Open Source projects is a plus * Familiarity with the Evergreen ILS is a plus This is a full-time salaried position. Benefits include full company-paid medical, dental, and vision insurance; paid sick and vacation time; and a 401k plan with matching company contribution. Equinox Software, Inc. is located in Norcross, GA. Atlanta area locals preferred, but teleworking is possible for the right set of candidates. Evergreen (http://evergreen-ils.org/) is an enterprise-class, open-source, integrated library system. Please send résumé or c.v. with cover letter to careers@esilibrary.com. Please see http://esilibrary.com for more information about Equinox Software, Inc. -bill -- Bill Erickson | VP, Software Development & Integration | Equinox Software, Inc. / The Evergreen Experts | phone: 877-OPEN-ILS (673-6457) | email: erickson@esilibrary.com | web: http://esilibrary.com http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2668.html Thu, 25 Sep 2008 09:57:54 +0000 Where's the support for Unimarc? by Emanuele Venezia Hello everybody, I'm about to use MARC::Record for the first time, and I've found nothing on CPAN about Unimarc. Did I do a wrong search? I happened to read about something some times ago, but I don't remember where. I've tried to use it step by step by simply creating an empty record and filling it field by field while reading a file (line by line). It almost worked, but it seems USMARC has some functions that could ease my work if they existed for Unimarc. Can someone address me please? Another question: is there a way to import from an Aleph sequential file? Thanks in advance, Emanuele http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2667.html Tue, 16 Sep 2008 23:48:31 +0000 Re: MARC.pm - Perl extension to manipulate MAchine Readable Cataloging records. by Dan Scott 2008/9/9 <Bbirthisel@aol.com>: > Hi Jeffrey, > > In a message dated 9/9/2008 10:09:39 AM Eastern Daylight Time, > Jeffrey.Lewis@proquest.com writes: > >> I was looking over this perl module and am curious if there is an support >> for MARC to Unicode character conversions. A prompt response would be much >> appreciated. > > Unicode was not supported in the original MARC.pm and related code. We had > limited XML support and some character mappings to maintain compatibility with > the older MS-DOS based USMARC programs. I think the later MARC::Record family > of modules does have at least some Unicode support. > > The place to ask is the perl_for_librarians mailing list, so I have copied my > reply to them. In a nutshell, MARC8 to UTF8 (Unicode) conversions are supported by MARC::Charset (http://search.cpan.org/dist/MARC-Charset/) -- Dan Scott Laurentian University http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2666.html Tue, 09 Sep 2008 17:48:21 +0000 Re: MARC.pm - Perl extension to manipulate MAchine Readable Cataloging records. by Bbirthisel Hi Jeffrey, In a message dated 9/9/2008 10:09:39 AM Eastern Daylight Time, Jeffrey.Lewis@proquest.com writes: > I was looking over this perl module and am curious if there is an support > for MARC to Unicode character conversions. A prompt response would be much > appreciated. Unicode was not supported in the original MARC.pm and related code. We had limited XML support and some character mappings to maintain compatibility with the older MS-DOS based USMARC programs. I think the later MARC::Record family of modules does have at least some Unicode support. The place to ask is the perl_for_librarians mailing list, so I have copied my reply to them. -bill ************** Psssst...Have you heard the news? There's a new fashion blog, plus the latest fall trends and hair styles at StyleList.com. (http://www.stylelist.com/trends?ncid=aolsty00050000000014) http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2665.html Tue, 09 Sep 2008 14:38:04 +0000 Position announcement - Koha application developer (US or Canada) by Galen Charlton [Apologies for cross-posting] LibLime has a full-time Perl application developer position that we're looking to fill immediately. The responsibilities will center on adding new features to the Koha ILS (http://www.koha.org/), but may include working on other open-source library projects such as the &Dagger;biblios metadata editor (http://www.biblios.org/). Applicants should be based in the United States or Canada. REASONS YOU'LL LOVE WORKING AT LIBLIME 1. Our customers. Working at LibLime you'll have the opportunity to work directly with libraries who have a real vision for where they want their library technology to go, and the resources to take it there. 2. Flexible, distributed environment. We have a central management office, but most of our employees work from home offices. Hours are flexible; you can listen to your own music and work in your pajamas (if you want to!). 3. It's all Open Source. We're geeks about open source; every piece of code you create is licensed under the GPL. We tell our customers that means no vendor lock-in, but it applies to employees too -- your code follows you to your next job. 4. Competitive compensation and benefits package. Excellent group health plan, 401(k) retirement account with employer matching, and support for travel to professional development opportunities such as the Code4Lib conference. 5. Upward mobility. LibLime is a growing company with lots of opportunities to advance your career. QUALIFICATIONS 4+ years of application development experience in Perl including SQL and web services. Experience working with an existing codebase, including good refactoring skills. Experience on the command line in Unix/Linux environment. PREFERRED QUALIFICATIONS Domain knowledge in library workflow in one or more areas such as acquisitions, authority control, serials, cataloging, or circulation. Knowledge of library-speciﬁc standards/protocols such as MARC, MODS, Dublin Core, Z39.50, other Z39.* standards, NCIP, SIP2. JOB DESCRIPTION The complete job description and qualifications can be found at http://liblime.com/about/careers/software_engineer_koha HOW TO APPLY Send a resume and cover letter to careers@liblime.com. -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2664.html Mon, 08 Sep 2008 10:22:01 +0000 Re: How convert data from USMARC to MARCXML using MARC::Record and it's derivatives? MARC.pm vs. MARC::Record.pm by Dmitry Arsentiev Galen, I'm sorry, It works correctly. I made mistake with typing in the name of the source file. I've just changed it from 01al2008.mrf to 01al2007.mrf and script became ready. Thank you! 2008/9/2 Galen Charlton <galen.charlton@liblime.com>: > Hi, > > On Tue, Sep 2, 2008 at 5:33 AM, Dmitry Arsentiev <dmarsentev@gmail.com> wrote: >> The question #1: Would it be right to develop my scripts with MARC.pm >> and MARC::XML.pm, >> or I should to develop only with MARC::Record > > I'd recommend using MARC::Record and MARC::File::XML. > >> The question #2: If I should to develop MARC::Record-based script, >> can I easy convert from MARC to MARCXML, >> or I've to write handler for each field I got >> from MARC? > > It can be as easy as loading a MARC record into a MARC::Record object > (e.g., $marc), then using the output of $marc->as_xml() or > $marc->as_xml_record(). > > Regards, > > Galen > -- > Galen Charlton > VP, Research & Development, LibLime > galen.charlton@liblime.com > p: 1-888-564-2457 x709 > skype: gmcharlt > http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2663.html Wed, 03 Sep 2008 10:02:44 +0000 Re: How convert data from USMARC to MARCXML using MARC::Record and it's derivatives? MARC.pm vs. MARC::Record.pm by Dmitry Arsentiev Thank you for your forewarning, Bill. I hadn't grow up enough to get this error yet. 2008/9/2 Bill Dueber <dueberb@umich.edu>: > Note that if you're working with XML at the record level that > MARC::File::XML has a bug (the logic for the 'as_xml_record' method is > reversed with respect to outputting XML headers). > > See this: > > http://rt.cpan.org/Public/Bug/Display.html?id=34082 > > It won't affect simple conversion, but is worth knowing about and fixing. > > > On 9/2/08 11:04 AM, "Galen Charlton" <galen.charlton@liblime.com> wrote: > >> Hi, >> >> On Tue, Sep 2, 2008 at 5:33 AM, Dmitry Arsentiev <dmarsentev@gmail.com> wrote: >>> The question #1: Would it be right to develop my scripts with MARC.pm >>> and MARC::XML.pm, >>> or I should to develop only with MARC::Record >> >> I'd recommend using MARC::Record and MARC::File::XML. >> >>> The question #2: If I should to develop MARC::Record-based script, >>> can I easy convert from MARC to MARCXML, >>> or I've to write handler for each field I got >>> from MARC? >> >> It can be as easy as loading a MARC record into a MARC::Record object >> (e.g., $marc), then using the output of $marc->as_xml() or >> $marc->as_xml_record(). >> >> Regards, >> >> Galen > > -- > Bill Dueber > Library Systems Programmer > University Library, University of Michigan > 734.615.0412 > > http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2662.html Wed, 03 Sep 2008 09:44:27 +0000 Re: How convert data from USMARC to MARCXML using MARC::Record and it's derivatives? MARC.pm vs. MARC::Record.pm by Dmitry Arsentiev Thank you for your answer, Galen. Unfortunately, my simple script does not work. Here it is. ---------------------------------------------------------- #!/usr/bin/perl use MARC::Record; use MARC::Batch; use MARC::File::XML ( BinaryEncoding => 'utf8', RecordFormat => 'USMARC' ); my $filename = "01al2008.mrf"; my $batch = MARC::Batch->new('USMARC', $filename ); my $record = $batch->next(); print $record->as_xml()."\n"; ---------------------------------------------------------- The error returned by my script is: Can't call method "as_xml" on an undefined value at try.pl line 9. I got the same result with the as_record_xml() I suppose, $batch->next() does not return MARC::Record object. 2008/9/2 Galen Charlton <galen.charlton@liblime.com>: > Hi, > > On Tue, Sep 2, 2008 at 5:33 AM, Dmitry Arsentiev <dmarsentev@gmail.com> wrote: >> The question #1: Would it be right to develop my scripts with MARC.pm >> and MARC::XML.pm, >> or I should to develop only with MARC::Record > > I'd recommend using MARC::Record and MARC::File::XML. > >> The question #2: If I should to develop MARC::Record-based script, >> can I easy convert from MARC to MARCXML, >> or I've to write handler for each field I got >> from MARC? > > It can be as easy as loading a MARC record into a MARC::Record object > (e.g., $marc), then using the output of $marc->as_xml() or > $marc->as_xml_record(). > > Regards, > > Galen > -- > Galen Charlton > VP, Research & Development, LibLime > galen.charlton@liblime.com > p: 1-888-564-2457 x709 > skype: gmcharlt > http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2661.html Wed, 03 Sep 2008 09:37:57 +0000 Re: How convert data from USMARC to MARCXML using MARC::Record andit's derivatives? MARC.pm vs. MARC::Record.pm by Bill Dueber Note that if you're working with XML at the record level that MARC::File::XML has a bug (the logic for the 'as_xml_record' method is reversed with respect to outputting XML headers). See this: http://rt.cpan.org/Public/Bug/Display.html?id=34082 It won't affect simple conversion, but is worth knowing about and fixing. On 9/2/08 11:04 AM, "Galen Charlton" <galen.charlton@liblime.com> wrote: > Hi, > > On Tue, Sep 2, 2008 at 5:33 AM, Dmitry Arsentiev <dmarsentev@gmail.com> wrote: >> The question #1: Would it be right to develop my scripts with MARC.pm >> and MARC::XML.pm, >> or I should to develop only with MARC::Record > > I'd recommend using MARC::Record and MARC::File::XML. > >> The question #2: If I should to develop MARC::Record-based script, >> can I easy convert from MARC to MARCXML, >> or I've to write handler for each field I got >> from MARC? > > It can be as easy as loading a MARC record into a MARC::Record object > (e.g., $marc), then using the output of $marc->as_xml() or > $marc->as_xml_record(). > > Regards, > > Galen -- Bill Dueber Library Systems Programmer University Library, University of Michigan 734.615.0412 http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2660.html Tue, 02 Sep 2008 09:13:52 +0000 Re: How convert data from USMARC to MARCXML using MARC::Record and it's derivatives? MARC.pm vs. MARC::Record.pm by Galen Charlton Hi, On Tue, Sep 2, 2008 at 5:33 AM, Dmitry Arsentiev <dmarsentev@gmail.com> wrote: > The question #1: Would it be right to develop my scripts with MARC.pm > and MARC::XML.pm, > or I should to develop only with MARC::Record I'd recommend using MARC::Record and MARC::File::XML. > The question #2: If I should to develop MARC::Record-based script, > can I easy convert from MARC to MARCXML, > or I've to write handler for each field I got > from MARC? It can be as easy as loading a MARC record into a MARC::Record object (e.g., $marc), then using the output of $marc->as_xml() or $marc->as_xml_record(). Regards, Galen -- Galen Charlton VP, Research & Development, LibLime galen.charlton@liblime.com p: 1-888-564-2457 x709 skype: gmcharlt http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2659.html Tue, 02 Sep 2008 08:04:27 +0000 How convert data from USMARC to MARCXML using MARC::Record and it's derivatives? MARC.pm vs. MARC::Record.pm by Dmitry Arsentiev Hi. I need help to get perl way to convert USMARC to MARCXML. 1. I found MARC::XML http://search.cpan.org/~bbirth/MARC-XML-0.4/XML.pm This module seems to be able to convert MARC<=>XML 2. But in the tutorial for MARC::Record http://search.cpan.org/~mikery/MARC-Record-2.0.0/lib/MARC/Doc/Tutorial.pod I've read the "History of MARC on CPAN" and I've make conclusions: - MARC.pm is dead, it is not supported; - MARC::Record and it's derivatives (MARC::Batch, MARC::File) are supported and developed nowadays. The question #1: Would it be right to develop my scripts with MARC.pm and MARC::XML.pm, or I should to develop only with MARC::Record The question #2: If I should to develop MARC::Record-based script, can I easy convert from MARC to MARCXML, or I've to write handler for each field I got from MARC? Thanks. Dmitry http://www.nntp.perl.org/group/perl.perl4lib/2008/09/msg2658.html Tue, 02 Sep 2008 03:33:10 +0000 Requesting help with simple MARC::File::XML program -- SOLVED by Anne L. Highsmith Thanks to suggestions from Michael Kreyche & Bill Dueber, I have been able to solve the problem with using the "file" command on my usmarc to marcxml program. It was the perl version. We were running on 5.8.7, which threw the error. Once we upgraded to 5.8.8, program ran just fine. http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2657.html Tue, 29 Jul 2008 07:40:54 +0000 Requesting help with simple MARC::File::XML program by Anne L. Highsmith I have a file of usmarc records which I want to read into a program and print to a file as MARC xml. Here's my program so far: ######################################################### #!/usr/local/bin/perl use strict; use warnings; use MARC::Record; use MARC::Batch; use MARC::File::XML; my $infile = 'updated_dissertation_records'; my $file = MARC::File::XML->out('updated_dissertation_records.xml', 'UTF-8' ); my $batch = MARC::Batch->new( 'USMARC', $infile); for (my $i = 0; $i < 3; $i++) { my $record = $batch->next(); $file->write($record); } ######################################################### First question -- I can't get past the my $file = MARC::File::XML->out('updated_dissertation_records.xml', 'UTF-8' ); statement. When I run the program, that line gets the error: usage $fh->binmode([LAYER]) at /usr/local/perl/5.8/lib/site_perl/5.8.7/MARC/File/XML.pm line 195 Two -- does the "$file->write()" statement expect a MARC::Record object, an XML stream, or a marc record string? I've assumed a MARC::Record object. If it expects an XML stream, what is the best method for what I'm trying to do to get my marc record into an appropriate XML stream? http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2656.html Sun, 20 Jul 2008 20:20:05 +0000 Re: Biblio::Isis and character encoding by Dobrica Pavlinusic On Mon, Jul 14, 2008 at 09:14:50AM +0200, Emmanuel Di Pretoro wrote: > Hi, > > Currently I'm trying to convert an ISIS database to MARC21. So I use > Biblio::Isis and MARC::Record to do that. No problem with this conversion, > except for some weird character encoding problems. Some bibliographic > records are in written in french, and accentuated characters like 'é' are > display as '<82>'. > > I've tried to use some Encode::* modules (Encode, Encode::Guess, > Encode::Detec, Encode::First), but without success. > > Is there anybody who have this kind of problem? Is there a solution? Biblio::Isis doesn't have any support for encoding. It will return content with original encoding from ISIS. This is intentional, because our local encoding was really wired. In our project WebPAC (which was reason to write Biblio::Isis in the first place :-) we are using Encode's from_to and/or decode to convert our local encoding to utf-8 which MARC::Record (2.0 and newer) handles well. See http://webpac.us/ for documentation or this snippet: http://svn.rot13.org/index.cgi/webpac2/view/trunk/lib/WebPAC/Output/MARC.pm p.s. WebPAC(2) is really universal conversion tool (data mangler :-) but it might be overkill for your purpose (or not). It also includes several DSL [domain specific languages] based on perl to massage data before producing output. HTH. -- Dobrica Pavlinusic 2share!2flame dpavlin@rot13.org Unix addict. Internet consultant. http://www.rot13.org/~dpavlin http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2655.html Tue, 15 Jul 2008 02:41:26 +0000 RE: Biblio::Isis and character encoding by Doran, Michael D Hi Emmanuel, > I'm trying to convert an ISIS database to MARC21 What is the character set encoding of the data in the ISIS database? What is the desired character set encoding for the MARC21 records? I.e. MARC-8 or MARC Unicode(UTF-8)? If they are dissimilar character encodings, is the data undergoing a character set conversion? -- Michael # Michael Doran, Systems Librarian # University of Texas at Arlington # 817-272-5326 office # 817-688-1926 mobile # doran@uta.edu # http://rocky.uta.edu/doran/ > -----Original Message----- > From: Emmanuel Di Pretoro [mailto:edipretoro@gmail.com] > Sent: Monday, July 14, 2008 2:15 AM > To: perl4lib@perl.org > Subject: Biblio::Isis and character encoding > > Hi, > > Currently I'm trying to convert an ISIS database to MARC21. So I use > Biblio::Isis and MARC::Record to do that. No problem with this conversion, > except for some weird character encoding problems. Some bibliographic > records are in written in french, and accentuated characters like 'Ã©' are > display as '<82>'. > > I've tried to use some Encode::* modules (Encode, Encode::Guess, > Encode::Detec, Encode::First), but without success. > > Is there anybody who have this kind of problem? Is there a solution? > > Thanks in advance. > > Regards, > > Emmanuel Di Pretoro http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2654.html Mon, 14 Jul 2008 12:44:07 +0000 Biblio::Isis and character encoding by Emmanuel Di Pretoro Hi, Currently I'm trying to convert an ISIS database to MARC21. So I use Biblio::Isis and MARC::Record to do that. No problem with this conversion, except for some weird character encoding problems. Some bibliographic records are in written in french, and accentuated characters like 'é' are display as '<82>'. I've tried to use some Encode::* modules (Encode, Encode::Guess, Encode::Detec, Encode::First), but without success. Is there anybody who have this kind of problem? Is there a solution? Thanks in advance. Regards, Emmanuel Di Pretoro http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2653.html Mon, 14 Jul 2008 00:14:57 +0000 Problems installing Yaz 3.0.34 prior to installing pazpar2 by Christopher Morgan Sorry - This isn't a Perl issue - I have posted it to the Yaz list instead. - Chris I installed Yaz 3.0.34 today and I got the following configuration: YAZ Package: yaz YAZ Version: 3.0.34 Bugreport: yaz-help@indexdata.dk Source code location: . C Preprocessor: gcc -E C Preprocessor flags: C Compiler: gcc C Compiler flags: -g -O2 Linker flags: Linked libs: -L/usr/lib -lxslt -lxml2 -lz -lpthread -lm Host System Type: i686-pc-linux-gnu Install path: /usr/local Automake: ${SHELL} /home/webadmin/yaz-3.0.34/config/missing --run automake-1.10 Archiver: ar Ranlib: ranlib However, I got this error message: configure: WARNING: libEXSLT development libraries not found. Since I want to try installing pazpar2 -- and I believe these libraries are necessary to do that --(?), I uninstalled Yaz using "make uninstall", removed the remaining yaz directories, and then tried installing libxslt-devel and libxml2-devel via rpms, but was unable to. libxml2 seemed to install correctly, for example, but libxml2-devel wouldn't install, complaining that I hadn't installed libxml2 (!) When I tried to reinstall Yaz, I got this error message during make: marcdump.o(.text+0x12d): In function `marcdump_read_xml': /home/webadmin/yaz-3.0.34/util/marcdump.c:112: undefined reference to `xmlReaderForFile' collect2: ld returned 1 exit status make[1]: *** [yaz-marcdump] Error 1 make[1]: Leaving directory `/home/webadmin/yaz-3.0.34/util' make: *** [all-recursive] Error 1 I suspect something got corrupted, so I am going to reset the VPS and try again. Does anyone have any suggestions about which development libraries to install before installing Yaz, if you plan to install pazpar2? Are there any tarballs of the libraries available online rather than rpms? I'm also not clear about which versions of the development libraries to install -- there are many different versions available. Thanks! - Chris Morgan http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2652.html Thu, 10 Jul 2008 11:58:15 +0000 Problems installing Yaz 3.0.34 prior to installing pazpar2 by Christopher Morgan I installed Yaz 3.0.34 today and I got the following configuration: YAZ Package: yaz YAZ Version: 3.0.34 Bugreport: yaz-help@indexdata.dk Source code location: . C Preprocessor: gcc -E C Preprocessor flags: C Compiler: gcc C Compiler flags: -g -O2 Linker flags: Linked libs: -L/usr/lib -lxslt -lxml2 -lz -lpthread -lm Host System Type: i686-pc-linux-gnu Install path: /usr/local Automake: ${SHELL} /home/webadmin/yaz-3.0.34/config/missing --run automake-1.10 Archiver: ar Ranlib: ranlib However, I got this error message: configure: WARNING: libEXSLT development libraries not found. Since I want to try installing pazpar2 -- and I believe these libraries are necessary to do that --(?), I uninstalled Yaz using "make uninstall", removed the remaining yaz directories, and then tried installing libxslt-devel and libxml2-devel via rpms, but was unable to. libxml2 seemed to install correctly, for example, but libxml2-devel wouldn't install, complaining that I hadn't installed libxml2 (!) When I tried to reinstall Yaz, I got this error message during make: marcdump.o(.text+0x12d): In function `marcdump_read_xml': /home/webadmin/yaz-3.0.34/util/marcdump.c:112: undefined reference to `xmlReaderForFile' collect2: ld returned 1 exit status make[1]: *** [yaz-marcdump] Error 1 make[1]: Leaving directory `/home/webadmin/yaz-3.0.34/util' make: *** [all-recursive] Error 1 I suspect something got corrupted, so I am going to reset the VPS and try again. Does anyone have any suggestions about which development libraries to install before installing Yaz, if you plan to install pazpar2? Are there any tarballs of the libraries available online rather than rpms? I'm also not clear about which versions of the development libraries to install -- there are many different versions available. Thanks! - Chris Morgan http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2651.html Wed, 09 Jul 2008 12:57:58 +0000 Re: Problem installing MARC::Record 2.0.0 under perl 5.8.0 by Brad Baxter On Tue, Jul 8, 2008 at 4:11 PM, Christopher Morgan <morgan@acm.org> wrote: > So there is hope! But, yes, I see the need to get to 5.8.2 asap! > or 5.10.0 :-) http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2650.html Tue, 08 Jul 2008 14:00:37 +0000 RE: Problem installing MARC::Record 2.0.0 under perl 5.8.0 by Christopher Morgan Brian, Thanks very much. I'll try that version. - Chris -----Original Message----- From: Bryan Baldus [mailto:bryan.baldus@quality-books.com] Sent: Tuesday, July 08, 2008 2:31 PM To: Christopher Morgan; perl4lib@perl.org Subject: RE: Problem installing MARC::Record 2.0.0 under perl 5.8.0 On Tuesday, July 08, 2008 12:35 PM, Christopher Morgan wrote: >I am in the process of rebuilding my web site after a phishing site >break-in (yikes!). The site is fine now, and secure, but for some >reason I can't get MARC::Record-2.0.0 to install. I get an error >message saying that perl 5.8.2 is required, but that I only have perl >5.8.0. (And indeed I do have perl 5.8.0) But I'm pretty sure this version of MARC::Record *did* install under perl 5.8.0 that last time I tried.< MARC::Record 1.39_02 appears to be the latest version on CPAN that would work on 5.8.0. MARC::Record 2.x is incompatible with pre-5.8.2 versions of Perl due to Unicode-related changes. The change was announced in a Perl4Lib message "MARC::Record v2.0 RC1", sent Fri 5/20/2005 2:35 PM, by Ed Summers. [1] [1] <http://www.nntp.perl.org/group/perl.perl4lib/2005/05/msg2070.html> I hope this helps, Bryan Baldus bryan.baldus@quality-books.com eijabb@cpan.org http://home.inwave.com/eija http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2649.html Tue, 08 Jul 2008 13:26:21 +0000 RE: Problem installing MARC::Record 2.0.0 under perl 5.8.0 by Christopher Morgan >I sure hope you meant upgrading to Perl 5.8.2 (or higher) rather than downgrading to >MARC::Record 1.39_02. ;-) Michael, I wish (:->)! Unfortunately, I'm stuck with Perl 5.8.0 because my VPS (virtual private server) at Apollo doesn't offer 5.8.2 - yet. Oh well. I had been succesfully using MARC::Charset with MARC::Record 1.39_2 up until my web site implosion, and the Unicode was working fine, including the marc8 -> utf8 conversions from Marc records and their subsequent display in browsers. So there is hope! But, yes, I see the need to get to 5.8.2 asap! - Chris http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2648.html Tue, 08 Jul 2008 13:14:33 +0000 RE: Problem installing MARC::Record 2.0.0 under perl 5.8.0 by Christopher Morgan Bryan, Problem solved. At your suggestion, I installed MARC::Record 1.39_02, and it's working fine. I was wrong about having installed version 2.0 in the past. When I looked at my old notes, I saw that I wasn't able to install version 2.0 because of Perl 5.8.0. - which makes sense. That's what I get for leaving a tarball of Marc::Record 2.0 in my backup directory (:->) Thanks again for your help! - Chris http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2647.html Tue, 08 Jul 2008 12:52:20 +0000 RE: Problem installing MARC::Record 2.0.0 under perl 5.8.0 by Doran, Michael D Hi Chris, > I'll try that version. I sure hope you meant upgrading to Perl 5.8.2 (or higher) rather than downgrading to MARC::Record 1.39_02. ;-) This is just my un-asked for 2 cents, but I wouldn't stint on anything that will make the processing of Unicode-encoded text easier. Last December seemed to mark a tipping point for Unicode, both on the internet: "Just last December [2007] there was an interesting milestone on the web. For the first time, we found that Unicode was the most frequent encoding found on web pages, overtaking both ASCII and Western European encodings" [1] ...as well as for its use in MARC records: "To facilitate the movement of records between MARC-8 and Unicode environments, it was recommended for an initial period that the use of Unicode be restricted to a repertoire identical in extent to the MARC-8 repertoire. [...] however, such a restriction is no longer appropriate. The full UCS repertoire, as currently defined at the Unicode web site, is valid for encoding MARC 21 records subject only to the constraints described [in the current MARC 21 Specifications]." [2] -- Michael [1] The Official Google Blog: "Moving to Unicode 5.1" http://googleblog.blogspot.com/2008/05/moving-to-unicode-51.html [2] MARC 21 Specifications: Unicode Encoding Environment (revised December 2007) http://www.loc.gov/marc/specifications/speccharucs.html # Michael Doran, Systems Librarian # University of Texas at Arlington # 817-272-5326 office # 817-688-1926 mobile # doran@uta.edu # http://rocky.uta.edu/doran/ > -----Original Message----- > From: Christopher Morgan [mailto:morgan@acm.org] > Sent: Tuesday, July 08, 2008 2:12 PM > To: 'Bryan Baldus'; perl4lib@perl.org > Subject: RE: Problem installing MARC::Record 2.0.0 under perl 5.8.0 > > Brian, > > Thanks very much. I'll try that version. > > - Chris > > > -----Original Message----- > From: Bryan Baldus [mailto:bryan.baldus@quality-books.com] > Sent: Tuesday, July 08, 2008 2:31 PM > To: Christopher Morgan; perl4lib@perl.org > Subject: RE: Problem installing MARC::Record 2.0.0 under perl 5.8.0 > > On Tuesday, July 08, 2008 12:35 PM, Christopher Morgan wrote: > >I am in the process of rebuilding my web site after a phishing site > >break-in (yikes!). The site is fine now, and secure, but for some > >reason I can't get MARC::Record-2.0.0 to install. I get an error > >message saying that perl 5.8.2 is required, but that I only have perl > >5.8.0. (And indeed I do have perl > 5.8.0) But I'm pretty sure this version of MARC::Record *did* install > under > perl 5.8.0 that last time I tried.< > > MARC::Record 1.39_02 appears to be the latest version on CPAN that would > work on 5.8.0. MARC::Record 2.x is incompatible with pre-5.8.2 versions of > Perl due to Unicode-related changes. The change was announced in a > Perl4Lib > message "MARC::Record v2.0 RC1", sent Fri 5/20/2005 2:35 PM, by Ed > Summers. > [1] > > [1] <http://www.nntp.perl.org/group/perl.perl4lib/2005/05/msg2070.html> > > I hope this helps, > > Bryan Baldus > bryan.baldus@quality-books.com > eijabb@cpan.org > http://home.inwave.com/eija http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2646.html Tue, 08 Jul 2008 12:49:48 +0000 RE: Problem installing MARC::Record 2.0.0 under perl 5.8.0 by Christopher Morgan Brian, Thanks very much. I'll try that version. - Chris -----Original Message----- From: Bryan Baldus [mailto:bryan.baldus@quality-books.com] Sent: Tuesday, July 08, 2008 2:31 PM To: Christopher Morgan; perl4lib@perl.org Subject: RE: Problem installing MARC::Record 2.0.0 under perl 5.8.0 On Tuesday, July 08, 2008 12:35 PM, Christopher Morgan wrote: >I am in the process of rebuilding my web site after a phishing site >break-in (yikes!). The site is fine now, and secure, but for some >reason I can't get MARC::Record-2.0.0 to install. I get an error >message saying that perl 5.8.2 is required, but that I only have perl >5.8.0. (And indeed I do have perl 5.8.0) But I'm pretty sure this version of MARC::Record *did* install under perl 5.8.0 that last time I tried.< MARC::Record 1.39_02 appears to be the latest version on CPAN that would work on 5.8.0. MARC::Record 2.x is incompatible with pre-5.8.2 versions of Perl due to Unicode-related changes. The change was announced in a Perl4Lib message "MARC::Record v2.0 RC1", sent Fri 5/20/2005 2:35 PM, by Ed Summers. [1] [1] <http://www.nntp.perl.org/group/perl.perl4lib/2005/05/msg2070.html> I hope this helps, Bryan Baldus bryan.baldus@quality-books.com eijabb@cpan.org http://home.inwave.com/eija http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2645.html Tue, 08 Jul 2008 12:20:00 +0000 RE: Problem installing MARC::Record 2.0.0 under perl 5.8.0 by Bryan Baldus On Tuesday, July 08, 2008 12:35 PM, Christopher Morgan wrote: >I am in the process of rebuilding my web site after a phishing site break-in (yikes!). The site is fine now, and secure, but for some reason I can't get MARC::Record-2.0.0 to install. I get an error message saying that perl 5.8.2 is required, but that I only have perl 5.8.0. (And indeed I do have perl 5.8.0) But I'm pretty sure this version of MARC::Record *did* install under perl 5.8.0 that last time I tried.< MARC::Record 1.39_02 appears to be the latest version on CPAN that would work on 5.8.0. MARC::Record 2.x is incompatible with pre-5.8.2 versions of Perl due to Unicode-related changes. The change was announced in a Perl4Lib message "MARC::Record v2.0 RC1", sent Fri 5/20/2005 2:35 PM, by Ed Summers. [1] [1] <http://www.nntp.perl.org/group/perl.perl4lib/2005/05/msg2070.html> I hope this helps, Bryan Baldus bryan.baldus@quality-books.com eijabb@cpan.org http://home.inwave.com/eija http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2644.html Tue, 08 Jul 2008 11:35:31 +0000 Problem installing MARC::Record 2.0.0 under perl 5.8.0 by Christopher Morgan I am in the process of rebuilding my web site after a phishing site break-in (yikes!). The site is fine now, and secure, but for some reason I can't get MARC::Record-2.0.0 to install. I get an error message saying that perl 5.8.2 is required, but that I only have perl 5.8.0. (And indeed I do have perl 5.8.0) But I'm pretty sure this version of MARC::Record *did* install under perl 5.8.0 that last time I tried. I cheated by changing line 2 in the Makefile.PL file to read "require perl-5.8.0" instead of "5.8.2". It installed, but it only passed about 20% of the tests during make test. Am I asking for trouble here? Will it work, or should I try installing an earlier version? (If so, which earlier version, and where should I get it?) Also, I saw a patch somewhere that you could use if you're installing into systems that use Perl 5.00xxx or earlier (or something to that effect). Any thoughts from anyone on this? Many thanks! - Chris http://www.nntp.perl.org/group/perl.perl4lib/2008/07/msg2643.html Tue, 08 Jul 2008 10:37:42 +0000 Cleaning MARC file by Emmanuel Di Pretoro Hi, Is there anybody who is already involved in the process of cleaning a MARC file. This means: - fusion multiple records into one single record; - or keep one record, and delete the others. Can you describe your methodology, as well as used algorithms. Thanks in advance. Regards, Emmanuel Di Pretoro http://www.nntp.perl.org/group/perl.perl4lib/2008/06/msg2642.html Thu, 26 Jun 2008 02:04:10 +0000 The Berman Catalog by md I have the raw data files of the former Hennepin County Library catalog and authority files. This is the innovative, unique catalog created by Sandy Berman. 1970s-2002. I would like to import the data into a MYSQL database. I assume this can be done with Perl, but don't know if an existing parser would work or if a custom program would be needed. I have no programming skills. There must be someone... here who knows and values Berman's work and is ready, willing and able to devote their knowledge and skills to making it accessible once again. Please contact me with questions on or off list. Thank You! Madeline Douglass mdougla@pclink.com http://www.sanfordberman.org http://www.nntp.perl.org/group/perl.perl4lib/2008/06/msg2641.html Sun, 22 Jun 2008 17:56:19 +0000 RE: Practicality of using DB_File on a Perl-based book site? by Christopher Morgan Harrison, That's useful information. Yes, I'll only be doing lookups, which simplifies things quite a bit. Given what you said about the Movable Type software, I assume DB_FILE would be a good way to keep track of website user names, passwords, cookies, and the like? - Chris -----Original Message----- From: vagrantscholar@gmail.com [mailto:vagrantscholar@gmail.com] On Behalf Of Harrison Dekker Sent: Friday, June 20, 2008 2:19 PM To: Christopher Morgan Subject: Re: Practicality of using DB_File on a Perl-based book site? Chris, I'm no expert, but it seems to me, that there should be less overhead using Berkeley DB compared to a relational DB, assuming that all you're doing is lookups. If you've got a bunch of post processing going on involving multiple large retrieval sets then you'll probably lose that edge, but that's only because your perl code would be doing the work that a more optimized SQL engine could be doing. SQL doesn't give you any improvement, however, when all you're doing is a key/value type lookup. Movable Type blog software uses BDB, at least it did in the past, and as far as I know it's quite reliable/scalable. I use BDB for one web servicey type application and I do have to throttle my requests if I'm sending them in batch, but the db isn't the bottleneck, it's apache or the php xml functions I use. -Harrison On Fri, Jun 20, 2008 at 10:42 AM, Christopher Morgan <morgan@acm.org> wrote: > > I'm designing a web site that will display MARC authority files > onscreen. I use a Perl hash that's tied to a (read-only) Berkeley > DB_file, and it works nicely. How practical is this approach if > there's going to be moderate traffic on a site? > > My DB_FILE is about 200MB, but of course Perl brings only small pieces > of the database into memory at any one time. Would the site bog down > if people were accessing records at the rate of, say, every few > seconds? Should I consider mySQL instead? I'd prefer to stick to > DB_FILE, since it's so easy and elegant -- and I can easily create complex data structures. > > What if one of my data files was significantly bigger (say, a GB or > two of MARC book records)? I don't have a feel for the pros and cons > of the various approaches to accessing large databases using Perl, but > tied hashes are pretty fast! In any case, I know I'll have to lock the > file during each read, via "flock" or the like. I haven't tried implementing the latter yet. > > Does anyone have any ideas about this? Are there other Perl forums I > should investigate regarding this topic? > > Many thanks! > > - Chris Morgan > > -- Harrison Dekker -- Coordinator of Data Services -- UC Berkeley Libraries 510-642-8095 :: GTalk:vagrantscholar :: AIM:hdekker :: Meebo:ucbdekker http://sunsite.berkeley.edu/wikis/datalab/ ------------------------ Q: Why is this email 5 sentences or less? A: http://five.sentenc.es http://www.nntp.perl.org/group/perl.perl4lib/2008/06/msg2640.html Fri, 20 Jun 2008 11:40:08 +0000 Practicality of using DB_File on a Perl-based book site? by Christopher Morgan I'm designing a web site that will display MARC authority files onscreen. I use a Perl hash that's tied to a (read-only) Berkeley DB_file, and it works nicely. How practical is this approach if there's going to be moderate traffic on a site? My DB_FILE is about 200MB, but of course Perl brings only small pieces of the database into memory at any one time. Would the site bog down if people were accessing records at the rate of, say, every few seconds? Should I consider mySQL instead? I'd prefer to stick to DB_FILE, since it's so easy and elegant -- and I can easily create complex data structures. What if one of my data files was significantly bigger (say, a GB or two of MARC book records)? I don't have a feel for the pros and cons of the various approaches to accessing large databases using Perl, but tied hashes are pretty fast! In any case, I know I'll have to lock the file during each read, via "flock" or the like. I haven't tried implementing the latter yet. Does anyone have any ideas about this? Are there other Perl forums I should investigate regarding this topic? Many thanks! - Chris Morgan http://www.nntp.perl.org/group/perl.perl4lib/2008/06/msg2639.html Fri, 20 Jun 2008 10:43:19 +0000 Re: Can't parse MARC Authority XML files with mx: prefixes in their tags by Mike Rylander On Wed, Jun 18, 2008 at 1:12 PM, Christopher Morgan <morgan@acm.org> wrote: > Mike, > > I tried both of your suggested fixes (changing Name to LocalName, and > running the updated patch), but no luck. I still get no error messages in > the error log, but the program silently fails to print a report. If I > manually remove the "mx:" namespace strings from all the tags, I can process > the files with no problem. (So one quick fix would be to simply run these > records through a quick search and replace routine.) > > Regarding the problem name authority files. They're all available on the web > from OCLC's experimental name authority service, at > http://alcme.oclc.org/eprintsUK/index.html > > You enter an author name (I entered "Robert Benchley"). Then I clicked on > the first link at http://errol.oclc.org/laf/n50-7168.html > > Finally, I clicked on the second link ("XML Record") to get this link: > http://errol.oclc.org/laf/n50-7168.MarcXML All of these have the "mx:" > namespace notation in their tags. Thanks. I will see if I can fix this on my installation, but since the LocalName (only) change did not work for you I have suspicions about the particular XML parser that's being chosen for the SAX part on your system. The pure-perl parser (in some versions) did not support namespaces well, and expat can be quirky as well. I'll let you know what I find, and thanks for testing. -- Mike Rylander | VP, Research and Design | Equinox Software, Inc. / The Evergreen Experts | phone: 1-877-OPEN-ILS (673-6457) | email: miker@esilibrary.com | web: http://www.esilibrary.com http://www.nntp.perl.org/group/perl.perl4lib/2008/06/msg2638.html Wed, 18 Jun 2008 11:56:02 +0000 RE: Can't parse MARC Authority XML files with mx: prefixes in theirtags by Christopher Morgan Mike, I tried both of your suggested fixes (changing Name to LocalName, and running the updated patch), but no luck. I still get no error messages in the error log, but the program silently fails to print a report. If I manually remove the "mx:" namespace strings from all the tags, I can process the files with no problem. (So one quick fix would be to simply run these records through a quick search and replace routine.) Regarding the problem name authority files. They're all available on the web from OCLC's experimental name authority service, at http://alcme.oclc.org/eprintsUK/index.html You enter an author name (I entered "Robert Benchley"). Then I clicked on the first link at http://errol.oclc.org/laf/n50-7168.html Finally, I clicked on the second link ("XML Record") to get this link: http://errol.oclc.org/laf/n50-7168.MarcXML All of these have the "mx:" namespace notation in their tags. - Chris http://www.nntp.perl.org/group/perl.perl4lib/2008/06/msg2637.html Wed, 18 Jun 2008 10:13:03 +0000 Re: Can't parse MARC Authority XML files with mx: prefixes in their tags by Mike Rylander On Tue, Jun 10, 2008 at 1:18 PM, Christopher Morgan <morgan@acm.org> wrote: > Mike, > > Sorry. Since my last post, I did find out how to use the UNIX patch command, > and applied your patch to SAX.pm. My script still doesn't work, and there > are no error messages. My earlier script (which worked on the subject > authority file) now does not work, so I'm wondering if something in the > patch may be causing this. I have a backup of the SAX.pm file in any case. > Well, it turns out I left something out of the patch I sent before. In the end_element sub, the second line should be my $name = $element->{ LocalName }; instead of my $name = $element->{ Name }; If you would, you can just edit the installed version of the patched SAX.pm to test. The next thing to try would be to remove the namespace test, but leave the LocalName changes in place. Anecdotal evidence suggests that some of the more popular XML parsing engines, or at least the Perl bindings for them, have problems with namespaces. I've attached a (complete, arg!) patch that implements just the LocalName changes and would be applied to the original version of SAX.pm. If you don't have time to test all this that's fined, but if not would you be willing to send a couple of your problem records? Thanks Christopher, -- Mike Rylander | VP, Research and Design | Equinox Software, Inc. / The Evergreen Experts | phone: 1-877-OPEN-ILS (673-6457) | email: miker@esilibrary.com | web: http://www.esilibrary.com http://www.nntp.perl.org/group/perl.perl4lib/2008/06/msg2636.html Thu, 12 Jun 2008 06:54:22 +0000 RE: Can't parse MARC Authority XML files with mx: prefixes in theirtags by Christopher Morgan Mike, Many thanks. My apologies, but I've never applied a Perl patch before, so I'm not sure of the correct procedure. I did locate the SAX.pm file. - Chris -----Original Message----- From: Mike Rylander [mailto:mrylander@gmail.com] Sent: Tuesday, June 10, 2008 11:57 AM To: Christopher Morgan Cc: jtgorman@uiuc.edu; perl4lib@perl.org Subject: Re: Can't parse MARC Authority XML files with mx: prefixes in their tags On Mon, Jun 9, 2008 at 5:39 PM, Christopher Morgan <morgan@acm.org> wrote: > Jonathan, > > Many thanks. I get no errors on the command line or in the error log > when I run the script. The file just executes with no output. If you > have the time to run it, I've included the scriupt below, and have > attached the name authority record it tries to process: The problem is that the SAX parser is looking for the element Name instead of LocalName. I've attached a patch that tests both LocalName and NamespaceURI. If you could apply this to your version of MARC/File/SAX.pm and give it a test, and it works for you, I'll commit it to the CVS repo. --miker http://www.nntp.perl.org/group/perl.perl4lib/2008/06/msg2635.html Tue, 10 Jun 2008 10:51:40 +0000 RE: Can't parse MARC Authority XML files with mx: prefixes in theirtags by Christopher Morgan Mike, Sorry. Since my last post, I did find out how to use the UNIX patch command, and applied your patch to SAX.pm. My script still doesn't work, and there are no error messages. My earlier script (which worked on the subject authority file) now does not work, so I'm wondering if something in the patch may be causing this. I have a backup of the SAX.pm file in any case. - Chris http://www.nntp.perl.org/group/perl.perl4lib/2008/06/msg2634.html Tue, 10 Jun 2008 10:20:17 +0000 Re: Can't parse MARC Authority XML files with mx: prefixes in their tags by Mike Rylander On Mon, Jun 9, 2008 at 5:39 PM, Christopher Morgan <morgan@acm.org> wrote: > Jonathan, > > Many thanks. I get no errors on the command line or in the error log when I > run the script. The file just executes with no output. If you have the time > to run it, I've included the scriupt below, and have attached the name > authority record it tries to process: The problem is that the SAX parser is looking for the element Name instead of LocalName. I've attached a patch that tests both LocalName and NamespaceURI. If you could apply this to your version of MARC/File/SAX.pm and give it a test, and it works for you, I'll commit it to the CVS repo. --miker > > #! /usr/bin/perl > use strict; > > use MARC::Record; > use MARC::Batch; > use MARC::File::XML; > use constant MAX => 20; > > MARC::File::XML->default_record_format('UNIMARCAUTH'); > my $batch = MARC::Batch->new( 'XML', 'name_authority_file'); > while (my $record = $batch->next()) { > for my $field ($record->field("100")){ > my $name= $field->subfield('a'); > print "$name", "\n"; > } > } > > I think you're right about the LOC files -- they probably got the extra > spaces by accident. That's easy enough to fix. > > As far as the name authorities go, if I can't get MARC::File::XML to process > them, I can always use XML::Tokeparser. Not as elegant, but it would get the > job done. > > - Chris > > -----Original Message----- > From: Jonathan Gorman [mailto:jtgorman@uiuc.edu] > Sent: Monday, June 09, 2008 4:43 PM > To: Christopher Morgan; perl4lib@perl.org > Subject: Re: Can't parse MARC Authority XML files with mx: prefixes in their > tags > > > >>However, I'm having trouble parsing the name authority records online >>at http://alcme.oclc.org/eprintsUK/index.html > > [snipped code examples] >> >>There are "mx:" prefixes in all the tags. What format is this? Is there >>any way I can get MARC::File::XML to parse these files? > > The prefixes are the namespace. The parser should be able to handle this, > but I don't honestly know if it does it correctly. What also might be the > problem is the second namespace in there. It might help us if you included > some information about what is not working (what error are you getting etc). > I don't have the time right now to run my own test, but actual error > messages might provide some clue. > >>A related question: When I first tried to process the subject authority >>files from the LOC (in my first example, above), the program complained >>that the "Leader must be 24 bytes long". > > Right, that comes from the MARC specification, there are 24 bytes. > >>XML files are five years old. I wonder if the XML spec has changed >>since >>then?) > > Doubt it, again it doesn't have anything really to do with the XML spec but > the underlying xml record. More likely it is some error in creating the > files. Can't give any more info though, sorry. > > Jon Gorman > -- Mike Rylander | VP, Research and Design | Equinox Software, Inc. / The Evergreen Experts | phone: 1-877-OPEN-ILS (673-6457) | email: miker@esilibrary.com | web: http://www.esilibrary.com http://www.nntp.perl.org/group/perl.perl4lib/2008/06/msg2633.html Tue, 10 Jun 2008 08:57:28 +0000 RE: Can't parse MARC Authority XML files with mx: prefixes in theirtags by Christopher Morgan Jonathan, Many thanks. I get no errors on the command line or in the error log when I run the script. The file just executes with no output. If you have the time to run it, I've included the scriupt below, and have attached the name authority record it tries to process: #! /usr/bin/perl use strict; use MARC::Record; use MARC::Batch; use MARC::File::XML; use constant MAX => 20; MARC::File::XML->default_record_format('UNIMARCAUTH'); my $batch = MARC::Batch->new( 'XML', 'name_authority_file'); while (my $record = $batch->next()) { for my $field ($record->field("100")){ my $name= $field->subfield('a'); print "$name", "\n"; } } I think you're right about the LOC files -- they probably got the extra spaces by accident. That's easy enough to fix. As far as the name authorities go, if I can't get MARC::File::XML to process them, I can always use XML::Tokeparser. Not as elegant, but it would get the job done. - Chris -----Original Message----- From: Jonathan Gorman [mailto:jtgorman@uiuc.edu] Sent: Monday, June 09, 2008 4:43 PM To: Christopher Morgan; perl4lib@perl.org Subject: Re: Can't parse MARC Authority XML files with mx: prefixes in their tags >However, I'm having trouble parsing the name authority records online >at http://alcme.oclc.org/eprintsUK/index.html [snipped code examples] > >There are "mx:" prefixes in all the tags. What format is this? Is there >any way I can get MARC::File::XML to parse these files? The prefixes are the namespace. The parser should be able to handle this, but I don't honestly know if it does it correctly. What also might be the problem is the second namespace in there. It might help us if you included some information about what is not working (what error are you getting etc). I don't have the time right now to run my own test, but actual error messages might provide some clue. >A related question: When I first tried to process the subject authority >files from the LOC (in my first example, above), the program complained >that the "Leader must be 24 bytes long". Right, that comes from the MARC specification, there are 24 bytes. >XML files are five years old. I wonder if the XML spec has changed >since >then?) Doubt it, again it doesn't have anything really to do with the XML spec but the underlying xml record. More likely it is some error in creating the files. Can't give any more info though, sorry. Jon Gorman http://www.nntp.perl.org/group/perl.perl4lib/2008/06/msg2632.html Mon, 09 Jun 2008 14:39:34 +0000