View source for Converting SFM Bibles to OSIS

==Introduction==

'''Standard Format Markers''' (SFM) and its more standardized derivative USFM have been used for decades to store Bibles for printing and display in programs like [http://paratext.ubs-translations.org/ UBS Paratext]. The format is popular among Bible translation agencies and Bible societies. The basic format of SFM is simply plaintext with backslash(\) codes. For example, a new Bible verse is signaled by '''\v''' followed by the number of the verse. 

Older texts in various dialects of SFM can still be found. It was common for various agencies, Bible societies, and even regional offices of such groups to have their own SFM standards. Unified Standard Format Markers (USFM) was developed to standardize SFM and encourage interoperability so that Bibles from one agency could be reasonably expected to operate with the software and stylesheets employed by another. The current version of USFM is 2.3, defined at http://confluence.ubs-icap.org/display/USFM/Home.

==Preparing (U)SFM files for conversion==

The simplicity of writing SFM also makes it easy to write poor SFM that fails to correspond to any kind of standard. The first task in preparing to convert SFM files to OSIS is to clean the text. The more regular your source files are, the more likely the conversion process will operate correctly.

One method of cleaning up your files is to import them into an SFM editor such as [http://sites.google.com/site/bibledit/ Bibledit] (which runs on Linux, Mac OS X, and Windows) or [http://www.sil.org/computing/fieldworks/BTE-FW_downloads.htm SIL FieldWorks Translation Editor] (which runs on Windows). The editors will frequently perform some basic corrections to the SFM syntax, but Bibledit in particular can perform a number of checks to correct specific errors common to SFM.

===Checking USFM files for versification issues===
Rather than waiting until you have the end result, a SWORD module, and then running the general utility '''emptyvss''', it is possible to use another Windows based utility called [http://gbcpreprocessor.codeplex.com/ GoBibleCreator USFM Preprocessor]. Despite its name, its use is not limited to conversion of documents for use towards making Go Bible applications. Although this was originally developed for supporting the Go Bible project, it does have some useful more general features. This is one of them, and is found under the "Check for Consistency" tab. Another is the ability to export the USFM Bible into BQ/DigiStudy format. The Search for Versification Issues is particularly helpful for Bibles structured primarily as "verse per line", rather than as paragraphs containing a verse range. Another useful feature is the identification of non-standard markers.

==Converting USFM files to OSIS==

This section gives details of software utilities capable of converting USFM files to OSIS:

=== usfm2osis.pl ===

:''It is usually advisable to use the most recent version of usfm2osis.pl &ndash; The version in [https://crosswire.org/svn/sword-tools/trunk/modules/perlconverters/ SVN] may be more recent than the copy in our ftpmirror''.

[http://crosswire.org/ftpmirror/pub/sword/utils/perl/usfm2osis.pl usfm2osis.pl] is a simple Perl script intended only for converting USFM files to OSIS. usfm2osis.pl was written by CrossWire, so its OSIS output is geared towards the use of OSIS documents in preparing to make SWORD modules. usfm2osis.pl is actively maintained by CrossWire, and bug reports and feature requests are welcomed at [mailto:sword-support@crosswire.org sword-support].

usfm2osis.pl requires a Perl interpreter in the system path. Then you can run: 
 perl usfm2osis.pl <osisWork> [-o OSIS-file] [-e USFM encoding] <USFM filenames|wildcard>

If usfm2osis.pl is not in the current directory, use its full path.

osisWork should be a value such as Bible.en.WEB.2007.

If you include an OSIS-file value, the output will be written there. Otherwise, it will be written to a file name based on your osisWork.

The USFM encoding argument should indicate the character encoding found in the source files. If none is given, UTF-8 is the default. The list of available encodings depends on your system. Executing the script with no arguments will print the list (as will executing it with an invalid encoding value).

The final argument is a list of filenames or a wildcard value such as *.sfm containing the SFM data. N.B. Such wildcards may not work in the Windows environment, due to the way the perl script interacts with the command shell.

In Windows, the following command line syntax will create an OSIS file for each SFM file. This assumes the current directory is the folder where the SFM files are stored, and that there is a same-level directory called osis. It also illustrates the use of (subsititute) drive p: for the full path to usfm2osis.pl, and omits the USFM encoding parameter merely for clarity.

 for %f in (*.SFM) do perl p:\usfm2osis.pl <osisWork> -o ..\osis\<osisWork>.%f.osis.xml %f

The additional use of %f within the output filenames is to give each OSIS XML file a unique name.

==== Fixing the output of usfm2osis.pl ====

<tt>usfm2osis.pl</tt> does not produce valid and working cross references. If your text contains such you need <tt>xreffix.pl</tt> which sits alongside <tt>usfm2osis.pl</tt> on the output of <tt>usfm2osis.pl</tt>. Using it requires the SWORD Perl bindings be installed and a sword-locale.

Unfortunately, (right now), the next step from a usfm2osis.pl file via osis2mod does not produce working titles. A further perl script, <tt>title-cleanup.pl</tt> adds the argument <tt>subType="x-preverse"</tt> to some titles.<ref>Also (right now) there is an unsolved issue that causes pre-verse titles to be misplaced.</ref>

Finally, and that has not much to do with <tt>usfm2osis.pl</tt>, many texts in non-Latin scripts (and some in Latin scripts) require batch conversion of characters and numbers. There are several scripts to assist with this, without harming the USFM or OSIS markup.

These supplementary Perl scripts are located in various sub-directories under [https://crosswire.org/svn/sword-tools/trunk/modules/ sword-tools].

'''Notes:'''
<references />

=== osis-converters ===

[http://code.google.com/p/osis-converters/ osis-converters] is the location for the open source (U)SFM-to-OSIS and OSIS-to-SWORD module converters developed independently by the main programmer of [http://code.google.com/p/xulsword/ xulsword]. A few of the OSIS elements used in these Perl scripts are not the same as generated by the CrossWire maintained scripts. See [http://code.google.com/p/osis-converters/wiki/Compatibility].

==Creation of publication ready modules from USFM and submission to CrossWire==
Before you import your OSIS files to SWORD format, you should check that it is valid OSIS.<BR>Before you submit any files to modules@crosswire.org, you '''must''' ensure that your files are valid OSIS. Invalid OSIS files will not be accepted.

Our process to create publication ready modules from USFM looks like this:

=== Stage 1 ===
''The command line examples in this section are for Unix. In Windows, it's not as simple to make equivalent command lines for Perl, especially in regard to [http://en.wikipedia.org/wiki/Glob_%28programming%29 globbing] wildcards for filenames''.
*Iterate usfm2osis.pl over each single USFM file to produce a  OSIS file per USFM file
 for biblebook in 'ls *usfm'; do usfm2osis.pl $biblebook $biblebook; done
*Check each OSIS file for XML validity - this will throw up a lot of not obvious USFM encoding problems
 for biblebook in 'ls *osis.xml'; do checkbible $biblebook; done
*Correct USFM files as per mistakes found

Rerun this stage until you have clean and validating OSIS files.

=== Stage 2 ===

# Check the OSIS files for not transformed USFM markers.<ref>Not many common ones are left unsupported, but you may encounter usually one or two new ones if you get files from a different new source.</ref>
# Fix usfm2osis.pl and add the missing tags for correct OSIS transformation.
# Rerun above until you have a clean validating collection of OSIS files with no left over USFM tags.<ref>A lot of the repetitive aspects can be done with the help of some shell scripts, so do not worry that you have to run each time multiples of usfm2osis.pl by hand.</ref>
# Send the updates to usfm2osis.pl to CrossWire's SVN repository.
# Run usfm2osis.pl over all the USFM files and create a single large OSIS file.
# Check again for XML validity.<ref>See [[OSIS_Bibles#Valid_OSIS_test|OSIS Bible validation]] for further instructions on OSIS validation.</ref>
# Run the OSIS  file through osis2mod<ref>Instructions for running osis2mod are available at [[osis2mod#Usage|osis2mod usage]].</ref>, create a [[DevTools:conf Files|conf file]], and check the resultant module for problems in a variety of front-ends.<ref>Fixes should be made in the USFM files at the very start and then cascaded down into the module following the overall process again.</ref>
# Create a [[DevTools:SWORD#Locale_file_layout|sword locale file]] and store in your <tt>/usr/share/sword/locales.d/</tt> directory<ref>or the equivalent directory on your system</ref> 
# If cross references are part of the USFM files fix these with the help of xreffix.pl (also in sword-tools).<ref>This requires a sword locale in the local installation at least.</ref>
# Run again through osis2mod and check again.
# Submit the final OSIS file and the conf file to CrossWire together with the locale file.<ref>Include advice regarding which [[Module Repositories|repository]] the module is intended for.</ref>
# Send any corrections on the USFM files back to the translation team. Chances are these are all valid and necessary corrections which help to improve the USFM.<ref>Both Paratext and Bibledit are a bit more forgiving than our tools, but some problems you encounter could have affected the final paper print.</ref>

'''Notes:'''
<references />

[[Category:Guides|Converting SFM Bibles to OSIS]]
[[Category:OSIS]]
[[Category:USFM]]
[[Category:Bibledit]]
[[Category:Paratext]]