Difference between revisions of "DevTools:Modules"

From CrossWire Bible Society
Jump to: navigation, search
(Brazilian)
m (Related Pages: deleted *GenBook and OpenOffice)
 
(186 intermediate revisions by 16 users not shown)
Line 1: Line 1:
= Module Repositories =
+
= Module Development Overview=
List of known [[Module Repositories]]
+
If you want to learn how to create a SWORD module, this is the place to start. Here is a brief overview of the process:
 +
#Collect and install the necessary software tools.
 +
#Obtain the source text and permission from the copyright holder if you wish to distribute a copyrighted module.
 +
#Prepare the source text for import.
 +
#Use an XML validator to check that your source file is properly constructed.
 +
#Import the source text using the appropriate tool.
 +
#Create a .conf file.
 +
#Install and test that the module displays correctly in several of the SWORD front-end applications.
 +
#Submit your module to CrossWire for distribution.
  
= Module Development=
+
=Creating a module=
 +
==Collect and Install Software Tools==
 +
*Install SWORD and collect SWORD module creation tools. For Linux (and Mac?) many of the module creation tools come installed with SWORD, so if you aren't comfortable installing from source simply install from your distribution's repositories. For Windows, you will need to download the module creation utilities from [http://www.crosswire.org/ftpmirror/pub/sword/utils/win32/ http://www.crosswire.org/ftpmirror/pub/sword/utils/win32/].<ref>Unofficial builds that may be from more recent SVN revisions may be found [http://dl.thehellings.com/sword-utils/ here].</ref> Be sure to download the most recent icudt dll and unzip it in the folder where you place the utilities.
  
== OSIS Book Name Abbreviations ==
+
*[[Frontends:Xiphos|Xiphos]] (for Windows) users should find a complete set of Sword tools in the Xiphos directory.
  
See [[DevTools:OSISBookNames]].
+
*The latest build of the Sword utilities for Windows requires the [http://www.microsoft.com/download/en/details.aspx?id=5555 Visual C++ 2010 Redistributable x86] version. If you have a 64-bit version of Windows, and you have installed only the [http://www.microsoft.com/download/en/details.aspx?id=14632 Visual C++ 2010 Redistributable x64] version, you will also need to install the x86 version.
  
== .conf File Layout ==
+
*Obtain a good programmer's text editor, preferentially one which does syntax and validation checks. See [[DevTools:Text Editors]] for examples
  
The conf file tell the Sword engine how to treat installed module files, etc. which kind of markup they contain, and so forth. Below is a listing of the possible directives in that file.
+
<references />
  
<table width="100%" border="1">
+
==Obtain Source Text and Permission to Distribute==
  <tr>
+
*The easiest texts to work with, especially for learning how to make a module, are texts in the [http://en.wikipedia.org/wiki/Public_domain public domain]. For example, you might try downloading a text from [http://www.ccel.org/ CCEL] first to get you started on the process of moving from a prepared text to a compiled module. Be sure to verify that the source text you obtain is indeed in the public domain.  
<td width="26%">
+
*Some people provide texts that are freely distributable under some sort of license (GNU GPL, Creative Commons, etc.) or no formal license. Be sure to document where the source provides that permission and check to see that they have the right to grant such permission so you can produce it if you want to distribute it with CrossWire.
<div align="center"><b>Element</b></div>
+
*For copyrighted material, you will need to contact the publisher or author to obtain permission. First check to see that someone else in CrossWire hasn't already pursued permission for that work. A list of requests and attempts to obtain rights on behalf of CrossWire can be found at [http://www.crosswire.org/wiki/index.php/Module_Requests Module Requests].
</td>
 
<td width="63%">
 
<div align="center"><b>Values (type or enumerated)</b></div>
 
</td>
 
<td width="11%">
 
<div align="center"><b>Default Value</b></div>
 
</td>
 
  </tr>
 
  <tr>
 
<td width="100%" colspan="3">
 
<div align="center"><font size="-1">required elements</font></div>
 
</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">DataPath</div>
 
</td>
 
<td width="63%">&lt;relative system path&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">Description</div>
 
</td>
 
<td width="63%">&lt;string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">ModDrv</div>
 
</td>
 
<td width="63%">RawText, zText, RawCom, zCom, HREFCom, RawFiles, RawLD, RawLD4,
 
zLD, RawGenBook</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="100%" colspan="3">
 
<div align="center"></div>
 
<div align="center"><font size="-1">elements required for proper module
 
  access</font></div>
 
</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">CipherKey</div>
 
</td>
 
<td width="63%">&lt;string&gt; (typically in a format matching the pattern
 
/[0-9]{4}[A-Za-z]{4}[0-9]{4}[A-Za-z]{4}/</td>
 
<td width="11%">
 
<div align="left"></div>
 
</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">BlockType</div>
 
</td>
 
<td width="63%">BOOK, CHAPTER, VERSE</td>
 
<td width="11%">CHAPTER</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">CompressType</div>
 
</td>
 
<td width="63%">ZIP, LZSS</td>
 
<td width="11%">LZSS</td>
 
  </tr>
 
  <tr>
 
<td width="26%">BlockCount</td>
 
<td width="63%">&lt;integer&gt;</td>
 
<td width="11%">200</td>
 
  </tr>
 
  <tr>
 
<td width="100%" colspan="3">
 
<div align="center"><font size="-1">elements required for proper rendering</font></div>
 
</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">GlobalOptionFilter</div>
 
</td>
 
<td width="63%">GBFStrongs, GBFFootnotes, GBFScripref, GBFMorph, GBFHeadings,
 
ThMLStrongs, ThMLFootnotes, ThMLScripref, ThMLMorph, ThMLHeadings, ThMLVariants,
 
ThMLLemma, UTF8Cantillation, UTF8GreekAccents, UTF8HebrewVowels</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">Direction</div>
 
</td>
 
<td width="63%">RtoL, LtoR</td>
 
<td width="11%">LtoR</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">SourceType</div>
 
</td>
 
<td width="63%">Plaintext, GBF, ThML, OSIS</td>
 
<td width="11%">Plaintext</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">Encoding</div>
 
</td>
 
<td width="63%">UTF-8, Latin-1</td>
 
<td width="11%">Latin-1</td>
 
  </tr>
 
  <tr>
 
<td width="26%">DisplayLevel</td>
 
<td width="63%">&lt;integer&gt;</td>
 
<td width="11%">1</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">Font</div>
 
</td>
 
<td width="63%">&lt;string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">OSISqToTick</div>
 
</td>
 
<td width="63%">true/false</td>
 
<td width="11%">true</td>
 
  </tr>
 
  <tr>
 
<td width="100%" colspan="3">
 
<div align="center"><font size="-1">elements to indicate features</font></div>
 
</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">Feature</div>
 
</td>
 
<td width="63%">StrongsNumbers, GreekDef, HebrewDef, GreekParse, HebrewParse,
 
DailyDevotion, Glossary</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">GlossaryFrom</div>
 
</td>
 
<td width="63%">&lt;OSIS:Lang identifier&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">GlossaryTo</div>
 
</td>
 
<td width="63%">&lt;OSIS:Lang identifier&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="100%" colspan="3">
 
<div align="center"><font size="-1">general informatic and installer elements</font></div>
 
</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">About</div>
 
</td>
 
<td width="63%">&lt;string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">Version</div>
 
</td>
 
<td width="63%">&lt;version string&gt;</td>
 
<td width="11%">1.0</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">History_x.x</div>
 
</td>
 
<td width="63%">&lt;string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">MinimumVersion</div>
 
</td>
 
<td width="63%">&lt;version string&gt;</td>
 
<td width="11%">1.5.1a</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">Category</div>
 
</td>
 
<td width="63%">&lt;string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">LCSH</div>
 
</td>
 
<td width="63%">&lt;tree/string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">Lang</div>
 
</td>
 
<td width="63%">&lt;OSIS:Lang identifier&gt;</td>
 
<td width="11%">en</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">InstallSize</div>
 
</td>
 
<td width="63%">&lt;integer (indicating bytes)&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">SwordVersionDate</div>
 
</td>
 
<td width="63%">&lt;ISO date string (yyyy-mm-dd)&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">Obsoletes</div>
 
</td>
 
<td width="63%">&lt;name of module&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="100%" colspan="3">
 
<div align="center"><font size="-1">copyright- &amp; licensing-related elements</font></div>
 
</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">CopyrightHolder</div>
 
</td>
 
<td width="63%">&lt;string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">CopyrightDate</div>
 
</td>
 
<td width="63%">&lt;integer (indicating year)&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">CopyrightNotes</div>
 
</td>
 
<td width="63%">&lt;string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">CopyrightContactName</div>
 
</td>
 
<td width="63%">&lt;string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">CopyrightContactAddress</div>
 
</td>
 
<td width="63%">&lt;string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">CopyrightContactEmail</div>
 
</td>
 
<td width="63%">&lt;string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">DistributionLicense</div>
 
</td>
 
<td width="63%">"Public Domain", "Copyrighted", "Copyrighted; Permission to distribute granted to CrossWire", "Copyrighted; Free non-commercial distribution"</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">DistributionSource</div>
 
</td>
 
<td width="63%">&lt;string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">DistributionNotes</div>
 
</td>
 
<td width="63%">&lt;string&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
  <tr>
 
<td width="26%">
 
<div align="left">TextSource</div>
 
</td>
 
<td width="63%">&lt;string, probably a URL&gt;</td>
 
<td width="11%">&nbsp;</td>
 
  </tr>
 
</table>
 
  
 +
==Prepare the Source Text for Import==
  
=Things that should move elsewhere=
+
===Versification===
 +
For Bible modules, SWORD supports a growing number of [[Alternate Versification|Versifications]]. A prerequisite for submission of a module that would not match the default versification for the KJV Bible is to decide which of the several versifications is most suitable for the new module.
  
== Module Requests ==
+
===Encoding===
 +
Note that the SWORD Project requires all submitted texts to be Unicode (UTF-8) encoded documents.
  
Here is a place to request modules you would like to be made. If the Copyright holder has been contacted, the permissions granted or not can be put here.  
+
Legacy texts might need [[DevTools:Conversion to Unicode|conversion to Unicode]].
  
====Bible Versions====
+
===Character sets===
*Nederlands Bijbelgenootschap [http://www.biblija.net/biblija.cgi?lang=nl], [http://80.252.86.246/]
+
A common problem in texts created by people less aware of Unicode principles is presence of characters which fit graphically but are out of set for this particular language. E.g. a Cyrillic a and a Latin a look identical but use different code points. It is important to clean this up prior to import as our module search depends on clean texts with consistent use of letters. Below a short shell script which will produce a list of all characters employed in a text.  
*Turkish Kutsal Kitap [http://www.kitabimukaddes.com/] [http://www.translation-trust.org/html/home.html]
 
*Cherokee New Testament [http://cherokeenewtestament.com/] '''Public Domain'''
 
*German revidierte Luther Bible 1984 [http://www.dbg.de]
 
  
====Books====
+
<pre>
 +
cat *txt |\
 +
uni2ascii -paU |\
 +
sed  -e "s/u/\nu/g" -e '/\./d' |\
 +
sort |\
 +
uniq -c -w5 |\
 +
sed -e 's/\\//' -e 's/\W+/\t/g' -e 's/\s*\([0-9][0-9]*\).*u\([0-9A-F]*\)/Character \\x\2 (u\2) was used \1 times/' |\
 +
ascii2uni -aB |\
 +
grep 'used' > charactercount.txt
 +
</pre>
  
====Devotionals====
+
===Images===
 +
Images can be included in any type of module. The specifics of how to do this is dependent upon markup.
  
====Lexicons====
+
Some SWORD applications are able to use virtually any image format, but SWORD only offically supports JPEG and PNG image files.
  
== New Modules ==
+
===Markup===
List of modules that are being developed
 
=== General Book ===
 
==== English ====
 
<ul>
 
<li>[[Creeds of Christendom by Philip Schaff]] ([http://www.ccel.org/ccel/schaff/creeds1.html 1] and [http://www.ccel.org/ccel/schaff/creeds2.html 2])</li>
 
</ul>
 
  
=== Portuguese ===
+
(''see also [[DevTools:Misc]]'')
==== Brazilian ====
 
SBB denied permissions, someone from Brasília intends to try again.
 
  
SBT has been contacted, anticipating answer.
+
We will accept only plain texts or texts marked up in OSIS or TEI, with the sole exception texts based on CCEL documents that are marked up in ThML.  
  
IBB has left the door open to a future favourable answer. Need to follow up with request for permissions on Versão Revisada and its Almeida Século XXI successor.
+
Internally, SWORD can process text in OSIS, TEI and 2 legacy formats(ThML, and GBF). From these formats, it can convert to other formats, including RTF and HTML, for display. OSIS 2.1 is now the preferred format for Bibles and commentaries. At the moment OSIS does not have thorough support for complex dictionaries. For that reason we support TEI for dictionaries.
  
[[Corrigida]] low-quality copies are available, we need to evaluate if it is ƿorþƿhile to move forward: old translation, not too good, OCR will be troublesome.
+
You may find documentation for each of these standards at their respective websites:<br/>
 +
*Open Scriptural Information Standard (OSIS) : http://www.bibletechnologies.net/<br/>
 +
*Text Encoding Initiative (TEI) : http://www.tei-c.org/P4X/DI.html<br/>
  
[[Tradução Brazileira]] copy obtained, working library contacts to find a book scanner.
+
In SWORD, for modules encoded  in OSIS, each verse, dictionary entry, and book division needs to be well-formed XML or it will result in display problems in some front-ends.
  
For all these, please ask for further information at sword-devel.
+
===Import formats===
  
==== Iberian ====
+
====OSIS Formatted General Books====
Permissions were obtained on a number of texts from [[SBP]], as per post at sword-devel.
+
With OSIS formatted general books, provided your document is well-formed and valid XML according to the OSIS 2.1 Schema, you should not need to do any further processing. You can use your XML file with xml2gbs. For OSIS encoded Bibles use [[osis2mod]].
 +
 
 +
====VPL Format====
 +
[[File Formats#VPL|VPL]] or verse-per-line format may only be used in creating Bibles. This format requires that each line start with a verse reference that SWORD can understand, such as "Genesis 1:1" or "Jn 3:16". Most English abbreviations are acceptable. Following the verse reference, the verse itself should be written, in any kind of markup. For example:
 +
 
 +
Genesis 1:1 In the beginning God created the heaven and the earth.
 +
Genesis 1:2 And the earth was without form, and void; and darkness was upon the face of the deep. And the Spirit of God moved upon the face of the waters.
 +
 
 +
This format is used with the utility vpl2mod, discussed below. To import Bibles that have have combined verses, you will need to use imp format, instead of vpl.
 +
 
 +
'''For CrossWire import purposes VPL is acceptable for text only Bibles without any further markup'''
 +
 
 +
===Imp Format ===
 +
 
 +
GenBook and LexDict modules may also be submitted in [[DevTools:Imp Format|Imp Format]].
 +
 
 +
==Validate the Source Text==
 +
 
 +
If your source text is either OSIS or TEI, you should definitely check it with a suitable XML text editor before proceeding to the next step.
 +
# Check that it has valid XML syntax
 +
# Validate the XML contents against the specified XML schema
 +
 
 +
==Import the Source Text==
 +
Now that your text is ready to be imported, you will need to use one of the command line utilities for converting documents to SWORD format. Depending on the format of your document at this point, you will need to use the appropriate importer.
 +
*If your text is a valid OSIS Bible, use [[osis2mod]].
 +
*If your text is a valid OSIS Commentary, use [[osis2mod]].
 +
*If your text is a valid OSIS document of some other type, use xml2gbs.
 +
*If your text is a valid TEI P5 dictionary, use [[TEI Dictionaries|tei2mod]].
 +
*If your text is a vpl format Bible, use vpl2mod.
 +
*If your text is an imp format Bible or commentary, use imp2vs.
 +
*If your text is an imp format dictionary, lexicon, glossary, or daily devotional, use imp2ld.
 +
*If your text is an imp format general book, use imp2gbs.
 +
 
 +
You may find these files in the SWORD Project source distribution or compiled for Win32 at http://crosswire.org/ftpmirror/pub/sword/utils/win32/. Each utility has brief usage information that can be viewed by running it once without any arguments.
 +
 
 +
===Compressing Modules===
 +
To compress a Bible, commentary, or LD module, use the mod2zmod utility. First you will need to install the module so that it can be accessed using the SWORD engine. Next, run:
 +
mod2zmod &lt;modname&gt; &lt;datapath&gt; [blockType [compressType]]
 +
blockType can be 4 = book (default), 3 = chapter, or 1 = verse and indicates the granularity of the compression blocks. The larger the block is, the longer it will take to access a piece of the text, but the smaller the resulting module will be. compressType can be either 1 = LZSS (default) or 2 = Zip.
 +
 
 +
You may wish to try different compression settings to find out which is best for your module. Typically, we use chapter compression for large commentaries, book compression for Bibles, and the Zip compression algorithm.
 +
 +
===[[Copyright#Locked_Modules|Locking Modules]]===
 +
To lock a rawText Bible or rawCom commentary module, use the cipherraw utility. Just run:
 +
cipherraw &lt;/path/to/module&gt; '&lt;key&gt;'
 +
 
 +
===Miscellaneous tools===
 +
Further miscellaneous tools that are 'not ready for public consumption' but may be useful to modules authors are found in [[DevTools:Misc]].  These includes scripts and programs that are used for the preparation and conversion of various specific modules.
 +
 
 +
===Debugging modules===
 +
 
 +
If you are not the module creator and thus do not have a copy of the source text file, some of the SWORD utilities can be used for help in debugging a module. It is especially useful to note that mod2imp followed by [imp2vs | imp2ld | imp2gbs] (depending on module type) often produces a lossless round-trip, no matter what the markup variety was used to make the original module. Editing the IMP file in between these two steps can therefore be used as a method to evaluate sensible conjectures regarding the apparent cause of a bug in a particular module.
 +
 
 +
:'''N.B.''' mod2imp followed by imp2(vs|ld|gbs) is not guaranteed to round-trip losslessly. mod2imp will give an accurate record of the contents of each content entry in the module, but skips past link entries. So for investigating entry contents in order to track down bugs, this process is fine, but CrossWire would never release content that had gone through this process.
 +
 
 +
If you have created a 'plain text' Bible module using either imp2vs or vpl2mod, it can be worthwhile to see what the module outputs with mod2osis. If the resulting osis file fails to pass the XML syntax check when viewed with a suitable XML editor, this may point to unexpected residual items in the source text. The proper solution then would be to raise the matter with the provider of the source text.
 +
 
 +
If you have created a devotional, imp2ld appears to have a bug that prevents it from creating the module correctly unless you use the trailing argument 2. 
 +
For example:
 +
imp2ld MyLD.txt MyLD 2
 +
 
 +
==Create the .conf File==
 +
In order to test and before submitting a new module, you need to create a .conf file, which tells Sword how to recognize and what to do with your module. Instructions for creating a .conf file are on the [[DevTools:conf Files]] page.
 +
 
 +
==Install and Test the Module==
 +
===Install the Module===
 +
Once you have imported the source document as a binary, several files will result. The number of files depends on the type of module, but you should make sure all of them are in the same folder. That folder will go in the modules folder, and the .conf file should go in the mods.d folder wherever your SWORD modules are installed. Open a front-end and see that it is recognizing the presence of the module. If it appears then you have installed it in the right place. Check to make sure that the content appears, and you are ready to start testing.
 +
 
 +
===Checking for Missing Verses===
 +
You can use the utility emptyvss to find verses in a module that contain no text, since this may indicate errors in the module. Just run:
 +
emptyvss &lt;module name&gt;
 +
on an installed module to generate a list.
 +
 
 +
==Submit the Module to CrossWire for Distribution==
 +
 
 +
First ensure that your module complies with our stated [[copyright]] policy.
 +
 
 +
If your module is to be sold rather than distributed for free, CrossWire has a tool to encrypt a module. However, we do not handle any payments, so we always suggest that it's for the owner (or an authorised agent) to host the payment system as well as the delivery of the unlock key.
 +
 
 +
After you have tested your module, you may wish to submit it to the SWORD Project for public release so that other people can benefit from your work. The submission itself should be of an uncompiled, plain text document in either VPL (verse-per-line), IMP (import),  OSIS or TEI [[File Formats|format]], ready to be run through one of the module build tools. '''Do not''' submit built modules that you have imported to Sword format; submit only source documents. You also need to supply the relevant conf file entries.  Before any module will be considered for hosting, we require that the following minimum set of module configuration fields be included in its .conf file:  '''Description''', '''About''', '''DistributionLicense''', and '''TextSource'''. For further detail please read [[Module Submission]]
 +
 
 +
When you decide that your module is ready to be submitted, you may email it to modules@crosswire.org. If you are unable to email it or would prefer to send the files by some other means, you may contact us at the same email address, and we can discuss other arrangements.
 +
 
 +
=Related Pages=
 +
*A Basic [[OSIS Tutorial]]
 +
*Guide to Writing [[OSIS Bibles]]
 +
*[[OSIS Book Abbreviations|OSIS Book Name Abbreviations]]
 +
*Guide to Writing [[OSIS Commentaries]]
 +
*Guide to Writing [[TEI Dictionaries]]
 +
*Guide to [[Converting SFM Bibles to OSIS]]
 +
*[[Encoding| Text Encoding]]
 +
*Related [[File Formats]]
 +
*List of [[Official and Affiliated Module Repositories]]
 +
*[[Zipped modules]]
 +
*[[Module Requests]]
 +
*[[Copyright]]
 +
 
 +
[[Category:Development tools|Modules]]
 +
[[Category:Modules]]

Latest revision as of 20:31, 4 June 2020

Module Development Overview

If you want to learn how to create a SWORD module, this is the place to start. Here is a brief overview of the process:

  1. Collect and install the necessary software tools.
  2. Obtain the source text and permission from the copyright holder if you wish to distribute a copyrighted module.
  3. Prepare the source text for import.
  4. Use an XML validator to check that your source file is properly constructed.
  5. Import the source text using the appropriate tool.
  6. Create a .conf file.
  7. Install and test that the module displays correctly in several of the SWORD front-end applications.
  8. Submit your module to CrossWire for distribution.

Creating a module

Collect and Install Software Tools

  • Install SWORD and collect SWORD module creation tools. For Linux (and Mac?) many of the module creation tools come installed with SWORD, so if you aren't comfortable installing from source simply install from your distribution's repositories. For Windows, you will need to download the module creation utilities from http://www.crosswire.org/ftpmirror/pub/sword/utils/win32/.[1] Be sure to download the most recent icudt dll and unzip it in the folder where you place the utilities.
  • Xiphos (for Windows) users should find a complete set of Sword tools in the Xiphos directory.
  • Obtain a good programmer's text editor, preferentially one which does syntax and validation checks. See DevTools:Text Editors for examples
  1. Unofficial builds that may be from more recent SVN revisions may be found here.

Obtain Source Text and Permission to Distribute

  • The easiest texts to work with, especially for learning how to make a module, are texts in the public domain. For example, you might try downloading a text from CCEL first to get you started on the process of moving from a prepared text to a compiled module. Be sure to verify that the source text you obtain is indeed in the public domain.
  • Some people provide texts that are freely distributable under some sort of license (GNU GPL, Creative Commons, etc.) or no formal license. Be sure to document where the source provides that permission and check to see that they have the right to grant such permission so you can produce it if you want to distribute it with CrossWire.
  • For copyrighted material, you will need to contact the publisher or author to obtain permission. First check to see that someone else in CrossWire hasn't already pursued permission for that work. A list of requests and attempts to obtain rights on behalf of CrossWire can be found at Module Requests.

Prepare the Source Text for Import

Versification

For Bible modules, SWORD supports a growing number of Versifications. A prerequisite for submission of a module that would not match the default versification for the KJV Bible is to decide which of the several versifications is most suitable for the new module.

Encoding

Note that the SWORD Project requires all submitted texts to be Unicode (UTF-8) encoded documents.

Legacy texts might need conversion to Unicode.

Character sets

A common problem in texts created by people less aware of Unicode principles is presence of characters which fit graphically but are out of set for this particular language. E.g. a Cyrillic a and a Latin a look identical but use different code points. It is important to clean this up prior to import as our module search depends on clean texts with consistent use of letters. Below a short shell script which will produce a list of all characters employed in a text.

 cat *txt |\
 uni2ascii -paU |\
 sed  -e "s/u/\nu/g" -e '/\./d' |\
 sort |\
 uniq -c -w5 |\
 sed -e 's/\\//' -e 's/\W+/\t/g' -e 's/\s*\([0-9][0-9]*\).*u\([0-9A-F]*\)/Character \\x\2 (u\2) was used \1 times/' |\
 ascii2uni -aB |\
 grep 'used' > charactercount.txt

Images

Images can be included in any type of module. The specifics of how to do this is dependent upon markup.

Some SWORD applications are able to use virtually any image format, but SWORD only offically supports JPEG and PNG image files.

Markup

(see also DevTools:Misc)

We will accept only plain texts or texts marked up in OSIS or TEI, with the sole exception texts based on CCEL documents that are marked up in ThML.

Internally, SWORD can process text in OSIS, TEI and 2 legacy formats(ThML, and GBF). From these formats, it can convert to other formats, including RTF and HTML, for display. OSIS 2.1 is now the preferred format for Bibles and commentaries. At the moment OSIS does not have thorough support for complex dictionaries. For that reason we support TEI for dictionaries.

You may find documentation for each of these standards at their respective websites:

In SWORD, for modules encoded in OSIS, each verse, dictionary entry, and book division needs to be well-formed XML or it will result in display problems in some front-ends.

Import formats

OSIS Formatted General Books

With OSIS formatted general books, provided your document is well-formed and valid XML according to the OSIS 2.1 Schema, you should not need to do any further processing. You can use your XML file with xml2gbs. For OSIS encoded Bibles use osis2mod.

VPL Format

VPL or verse-per-line format may only be used in creating Bibles. This format requires that each line start with a verse reference that SWORD can understand, such as "Genesis 1:1" or "Jn 3:16". Most English abbreviations are acceptable. Following the verse reference, the verse itself should be written, in any kind of markup. For example:

Genesis 1:1 In the beginning God created the heaven and the earth.
Genesis 1:2 And the earth was without form, and void; and darkness was upon the face of the deep. And the Spirit of God moved upon the face of the waters.

This format is used with the utility vpl2mod, discussed below. To import Bibles that have have combined verses, you will need to use imp format, instead of vpl.

For CrossWire import purposes VPL is acceptable for text only Bibles without any further markup

Imp Format

GenBook and LexDict modules may also be submitted in Imp Format.

Validate the Source Text

If your source text is either OSIS or TEI, you should definitely check it with a suitable XML text editor before proceeding to the next step.

  1. Check that it has valid XML syntax
  2. Validate the XML contents against the specified XML schema

Import the Source Text

Now that your text is ready to be imported, you will need to use one of the command line utilities for converting documents to SWORD format. Depending on the format of your document at this point, you will need to use the appropriate importer.

  • If your text is a valid OSIS Bible, use osis2mod.
  • If your text is a valid OSIS Commentary, use osis2mod.
  • If your text is a valid OSIS document of some other type, use xml2gbs.
  • If your text is a valid TEI P5 dictionary, use tei2mod.
  • If your text is a vpl format Bible, use vpl2mod.
  • If your text is an imp format Bible or commentary, use imp2vs.
  • If your text is an imp format dictionary, lexicon, glossary, or daily devotional, use imp2ld.
  • If your text is an imp format general book, use imp2gbs.

You may find these files in the SWORD Project source distribution or compiled for Win32 at http://crosswire.org/ftpmirror/pub/sword/utils/win32/. Each utility has brief usage information that can be viewed by running it once without any arguments.

Compressing Modules

To compress a Bible, commentary, or LD module, use the mod2zmod utility. First you will need to install the module so that it can be accessed using the SWORD engine. Next, run:

mod2zmod <modname> <datapath> [blockType [compressType]]

blockType can be 4 = book (default), 3 = chapter, or 1 = verse and indicates the granularity of the compression blocks. The larger the block is, the longer it will take to access a piece of the text, but the smaller the resulting module will be. compressType can be either 1 = LZSS (default) or 2 = Zip.

You may wish to try different compression settings to find out which is best for your module. Typically, we use chapter compression for large commentaries, book compression for Bibles, and the Zip compression algorithm.

Locking Modules

To lock a rawText Bible or rawCom commentary module, use the cipherraw utility. Just run:

cipherraw </path/to/module> '<key>'

Miscellaneous tools

Further miscellaneous tools that are 'not ready for public consumption' but may be useful to modules authors are found in DevTools:Misc. These includes scripts and programs that are used for the preparation and conversion of various specific modules.

Debugging modules

If you are not the module creator and thus do not have a copy of the source text file, some of the SWORD utilities can be used for help in debugging a module. It is especially useful to note that mod2imp followed by [imp2vs | imp2ld | imp2gbs] (depending on module type) often produces a lossless round-trip, no matter what the markup variety was used to make the original module. Editing the IMP file in between these two steps can therefore be used as a method to evaluate sensible conjectures regarding the apparent cause of a bug in a particular module.

N.B. mod2imp followed by imp2(vs|ld|gbs) is not guaranteed to round-trip losslessly. mod2imp will give an accurate record of the contents of each content entry in the module, but skips past link entries. So for investigating entry contents in order to track down bugs, this process is fine, but CrossWire would never release content that had gone through this process.

If you have created a 'plain text' Bible module using either imp2vs or vpl2mod, it can be worthwhile to see what the module outputs with mod2osis. If the resulting osis file fails to pass the XML syntax check when viewed with a suitable XML editor, this may point to unexpected residual items in the source text. The proper solution then would be to raise the matter with the provider of the source text.

If you have created a devotional, imp2ld appears to have a bug that prevents it from creating the module correctly unless you use the trailing argument 2. For example:

imp2ld MyLD.txt MyLD 2

Create the .conf File

In order to test and before submitting a new module, you need to create a .conf file, which tells Sword how to recognize and what to do with your module. Instructions for creating a .conf file are on the DevTools:conf Files page.

Install and Test the Module

Install the Module

Once you have imported the source document as a binary, several files will result. The number of files depends on the type of module, but you should make sure all of them are in the same folder. That folder will go in the modules folder, and the .conf file should go in the mods.d folder wherever your SWORD modules are installed. Open a front-end and see that it is recognizing the presence of the module. If it appears then you have installed it in the right place. Check to make sure that the content appears, and you are ready to start testing.

Checking for Missing Verses

You can use the utility emptyvss to find verses in a module that contain no text, since this may indicate errors in the module. Just run:

emptyvss <module name>

on an installed module to generate a list.

Submit the Module to CrossWire for Distribution

First ensure that your module complies with our stated copyright policy.

If your module is to be sold rather than distributed for free, CrossWire has a tool to encrypt a module. However, we do not handle any payments, so we always suggest that it's for the owner (or an authorised agent) to host the payment system as well as the delivery of the unlock key.

After you have tested your module, you may wish to submit it to the SWORD Project for public release so that other people can benefit from your work. The submission itself should be of an uncompiled, plain text document in either VPL (verse-per-line), IMP (import), OSIS or TEI format, ready to be run through one of the module build tools. Do not submit built modules that you have imported to Sword format; submit only source documents. You also need to supply the relevant conf file entries. Before any module will be considered for hosting, we require that the following minimum set of module configuration fields be included in its .conf file: Description, About, DistributionLicense, and TextSource. For further detail please read Module Submission

When you decide that your module is ready to be submitted, you may email it to modules@crosswire.org. If you are unable to email it or would prefer to send the files by some other means, you may contact us at the same email address, and we can discuss other arrangements.

Related Pages